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1.0  Purpose  and  Background 

1.1  Purpose 

This  report  describes  the  Joint  Advanced  Distributed  Simulation  Joint  Test  Force  (JADS  JTF)  test 
control  architecture.  It  outlines  the  test  control  design  requirements,  test  control  description,  and 
describes  the  physical  components.  Also,  this  report  addresses  JADS  JTF  concerns,  and  lessons 
learned.  It  is  intended  to  provide  insight  into  the  process  JADS  JTF  undertook  in  setting  up  a 
distributed  test  control  architecture  capable  of  supporting  distributed  testing. 

1.2  Background 

The  Joint  Advanced  Distributed  Simulation  Joint  Test  and  Evaluation  (JADS  JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),  Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  Advanced  Distributed  Simulation  (ADS)  technologies  for 
support  of  Developmental  Test  and  Evaluation  (DT&E)  and  Operational  Test  and  Evaluation 
(OT&E).  The  program  is  Air  Force  led,  with  Army  and  Navy  participation. 

The  JADS  JTF  is  directly  investigating  distributed  test  applications  in  three  slices  of  the  T&E 
spectrum:  A  Systems  Integration  Test  (SIT)  which  explores  distributed  testing  of  air-to-air 
missiles,  and  End-to-End  (ETE)  test  which  investigates  distributed  testing  of  command,  control, 
communications,  computers,  and  intelligence,  surveillance  and  reconnaissance  (C4ISR)  systems, 
and  Electronic  Warfare  (EW)  test  which  explores  distributed  testing  of  EW  systems. 

1.2.1  Systems  Integration  Test  (SIT)  Description 

The  SIT  evaluated  the  utility  of  using  distributed  testing  to  support  cost  effective  testing  of  an 
integrated  missile  weapon/launch  aircraft  system  in  an  operationally  realistic  scenario.  The  SIT 
also  evaluated  the  capability  of  the  JADS  Test  Control  and  Analysis  Center  (TCAC)  to  control  a 
distributed  test  of  this  type,  and  to  remotely  monitor  and  analyze  test  results.  The  SIT  consisted 
of  the  Linked  Simulators  Phase  (LSP)  and  the  Live  Fly  Phase  (LFP).  The  missions  simulated  a 
single  shooter  aircraft  launching  an  air-to-air  missile  against  a  single  target  aircraft.  The  LSP 
incorporated  a  manned  F-18  avionics  lab  (simulator)  at  China  Lake  NAS  as  the  shooter,  a  manned 
F-14  avionics  lab  (simulator)  at  Point  Mugu  NAS  as  the  target,  and  a  missile  hardware-in-the- 
loop  (HWIL)  simulation  lab  (simulator)  at  China  Lake  NAS  which  generated  AIM-9  missile  fly¬ 
outs  and  injected  countermeasures  (flares).  The  LFP  employed  an  architecture  which 
incorporated  a  live  F-16  shooter  aircraft,  a  live  F-16  target  aircraft,  and  an  Advanced  Medium 
Range  Air-to-Air  Missile  (AMRAAM)  HWIL  simulator  hosted  in  the  Eglin  AFB  Missile  Lab. 


1.2.2  End-to-End  (ETE)  Test  Description 
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The  ETE  test  was  designed  to  evaluate  the  utility  of  distributed  testing  to  support  testing  of 
C4ISR  systems.  The  test  used  the  developmental  and  operational  testing  issues  for  the  Joint 
Surveillance  Target  Attack  Radar  System  (Joint  STARS)  in  an  ADS-enhanced  environment  to 
conduct  its  T&E  utility  evaluation.  Also,  the  ETE  test  evaluated  the  capability  of  the  JADS 
TCAC  to  control  a  distributed  test  and  remotely  monitor  and  analyze  test  results.  The  ETE  test 
consisted  of  four  phases.  Phase  1  developed  or  modified  the  components  that  allowed  a  mix  of 
live  and  simulated  targets  at  an  E-8C  operator’s  console  and  a  light  ground  station  module 
(LGSM)  operator’s  console.  Phase  2  evaluated  the  utility  of  distributed  testing  to  support  DT&E 
and  early  OT&E  of  a  C4ISR  system  in  a  laboratory  environment.  Phase  3  moved  portions  of  the 
architecture  to  the  E-8C  aircraft,  ensured  that  the  components  functioned  properly,  and  confirmed 
the  synthetic  environment  interacted  properly  with  the  aircraft  and  actual  LGSM.  Phase  4 
evaluated  the  ability  to  perform  test  and  evaluation  of  the  E-8C  and  LGSM  in  a  synthetically 
enhanced  operational  environment,  using  typical  operators. 

1.2.3  Electronic  Warfare  (EW)  Test  Description 

The  EW  test  was  designed  to  evaluate  the  utility  of  a  distributed  EW  test  environment.  It 
consisted  of  three  phases.  Phase  1  consisted  of  open  air  range  and  hardware-in-the-loop  testing 
to  develop  a  performance  baseline  for  the  two  subsequent  phases.  Phase  2  employed  a  linked 
architecture  that  utilized  the  Department  of  Defense’s  (DOD)  high  level  architecture  (HLA)  and 
included  a  digital  system  model  of  the  ALQ-131  self-protection  jammer,  threat  simulation 
facilities,  and  constructive  models  that  replicated  the  open  air  environment.  Phase  3  substituted 
an  installed  systems  test  facility  (anechoic  chamber)  with  an  ALQ-131  pod  mounted  on  an  F-16 
for  the  digital  systems  model.  Both  phase  2  and  phase  3  compared  system  performance  data  with 
five  fly  data  from  phase  1  for  verification  and  validation  (V&V). 


2.0  Test  Control  Design  Requirements 

The  test  control  design  requirements  are  broken  into  four  distinct  areas:  common,  SIT,  ETE  and 
EW  requirements.  The  common  test  control  requirements  were  provided  by  the  JADS  Steering 
Committee/Leadership.  The  individual  test  team  requirements  were  provided  by  the  JADS  test 
teams. 

2.1  Common  Test  Control  Requirements 

Situational  awareness  of  a  test  event  is  critical  if  success  is  the  desired  result.  One  difficult 
challenge  facing  JADS  was  to  ensure  the  test  director  had  easy  access  to  all  of  the  information 
necessary  to  make  the  decisions  required  for  proper  test  execution.  With  complex  systems  at 
different  sites  interacting,  the  flow  of  information  in  near  real-time  requires  the  proper  equipment, 
tools,  and  people  be  placed  in  the  ideal  locations  and  the  available  information  be  easily 
transmitted  to  all  parties,  especially  to  the  test  director.  Procedures  were  developed  to  control  the 
flow  of  information  to  the  test  director.  For  the  test  director  too  much  information  is  just  as  bad 
or  worse  than  little  information.  Personnel  at  each  site  were  given  the  responsibility  of  providing 
specific  information  (system  status,  data  quality,  etc.)  at  predetermined  times  to  the  test  director. 
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Discipline  by  all  personnel  was  important.  Rehearsals  were  used  to  fine  tune  the  processes. 
Although  the  three  JADS  JTF  test  programs  were  investigating  the  utility  of  distributed  testing  in 
distinctly  different  environments,  the  following  were  common  requirements  between  the  tests: 

2.1.1  Voice  Communications 

Communications  are  common  to  any  form  of  testing  from  conventional  to  distributed  test  designs. 
All  JADS  tests  had  the  requirement  to  communicate  between  all  participants  located  in  various 
geographical  locations  .  Most  of  the  facilities  used  in  JADS  testing  had  implemented  intricate 
communication  networks  to  support  local  testing.  Connecting  these  networks  into  the  JADS 
communication  architecture  to  support  test  control  presented  JADS  with  a  few  challenges. 
Successful  distributed  test  control  communications  require  significant  design,  planning,  and 
rehearsal  efforts.  Communications  for  distributed  testing  can  range  from  the  simple  inexpensive 
solutions  such  as  commercial  phone  lines  to  encrypted  dedicated  communications  circuits  with 
larger  price  tags. 

2.1.2  Visual  Displays 

Since  all  three  JADS  test  involved  the  control  of  live  and/or  virtual  entities,  an  adequate  means  of 
visualizing  test  events  was  required..  Requirements  for  each  test  varied  in  complexity  but  the 
ability  to  observe  the  overall  scenario/engagement  was  needed  for  the  test  director  located  in  the 
TCAC.  The  use  of  the  visual  display  systems  was  required  for  system  integration,  test  rehearsal, 
and  test  execution.  Having  the  exact  same  view  of  the  test  space  at  all  sites  often  enhances  the 
ability  of  the  participants  to  interact  and  exchange  information  since  all  parties  are  seeing  the  same 
events  in  the  same  way.  The  translation  from  the  view  of  the  local  simulation  to  the  virtual  test 
space  often  introduces  errors.  When  multiple  sites  interact,  the  complex  behaviors  can  make  it 
difficult  to  detect  problems  if  everyone  has  their  own  way  of  viewing  the  virtual  world.  Use  of 
these  display  systems  enabled  JADS  test  team  to  visually  identify  problems  with  the  operation  and 
interaction  of  the  linked  simulations. 


2.1.3  Network  Monitoring 

Distributed  testing  requires  the  use  of  a  network  to  connect  the  test  nodes  together.  The  test 
nodes  used  during  JADS  testing  ranged  from  those  separated  by  over  a  thousand  miles  to  nodes 
separated  by  only  a  few  feet.  As  a  result,  the  test  director  needs  a  means  to  monitor  the  status  of 
the  network  links.  The  ability  to  monitor  the  network  in  real  time  allows  the  test  director  to 
control  activities  at  all  nodes  to  maximize  testing  efforts  and  minimize  expenditures  of  valuable 
assets.  For  the  JADS  testing  the  impact  of  network  losses  varied  from  a  total  loss  of  testing  to 
minor  degradation  of  test  objectives. 

2.1.4  Data  Transfer 
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Another  common  requirement  of  the  JADS  test  was  the  ability  to  transfer  large  amounts  of  data 
to  the  TCAC  from  other  test  nodes.  The  majority  of  the  data  transferred  was  used  for  detailed 
analysis  not  associated  with  test  control.  A  critical  portion  of  the  data  was  needed  to  support  the 
test  director  in  making  decisions  that  affected  the  conduct  of  a  current  test  or  the  scheduling  of 
the  next  test  event.  Rapid  access  to  this  data  was  key  in  preventing  the  expenditure  of  valuable 
test  time. 

2.1.5  Test  Control  Procedures 

The  development  and  use  of  detailed  test  control  procedure  was  common  to  all  JADS  test. 
Although  the  procedures  vary  from  test  to  test  the  goal  was  to  have  a  thorough  and 
practiced  method  to  control  the  test  event.  Development  of  these  procedures  were  included 
in  the  planning  process  from  the  start.  For  all  test,  JADS  personnel  used  risk  reduction 
and  integration  test  to  refine  draft  test  procedures.  This  refining  process  continued 
between  all  test  phases.  Development  of  test  control  procedures  included  key  members 
from  all  test  sites  to  ensure  an  understanding  of  site  requirements  and  existing  procedures. 

The  test  procedures  for  the  JADS  test  efforts  can  be  found  in  the  following  References: 

1 .  Systems  Integration  Test.  Linked  Simulators  Phase.  Final  Report.  Joint  Advanced  Distributed  Simulation  Joint 
Test  and  Evaluation,  Albuquerque,  New  Mexico,  July  1997 

2.  Systems  Integration  Test.  Live  Flv  Phase.  Final  Report.  Joint  Advanced  Distributed  Simulation  Joint  Test  and 
Evaluation,  Albuquerque,  New  Mexico,  March  1998 

3.  End-to-End  Test  Interim  Report  Phase  4.  Joint  Advanced  Distributed  Simulation  Joint  Test  and  Evaluation, 
Albuquerque,  New  Mexico,  August  1999 

4.  Electronic  Warfare  Test  Interim  Report  Phase  3.  Joint  Advanced  Distributed  Simulation  Joint  Test  and 
Evaluation,  Albuquerque,  New  Mexico,  November  1999. 


2.2  Systems  Integration  Test  (SIT)  Test  Control  Requirements 

The  SIT  test  control  requirements  differed  between  the  SIT  LSP  and  the  SIT  LFP  test  events. 
Test  control  had  to  support  the  following: 

2.2.1  Linked  Simulator  Phase  (LSP)  Test  Control  Requirements 

1 .  Unclassified  conference  call  network  to  support  up  to  10  phones  for: 

a.  Ordering  start/stop  of  each  simulation 

b.  Ordering  start/stop  of  data  loggers 

c.  Directing  simulation  pilots  engagement 

2.  Classified  STU-III  phone  capability  between  three  sites  (Pt  Mugu,  China  Lake 
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and  TCAC  for  discussion  of  classified  issues. 

3.  2-D  display  of  target/shooter/missile  engagement  in  the  TCAC. 

4.  Display  capability  of  target/shooter  dynamics  in  the  TCAC  (speed,  heading,  G’s, 
altitude,  range  to  target,  closing  speed,) 

5.  Quick-look  analysis  capability  to  verify  proper  execution  of  last  engagement. 

2.2.2  LSP  Network  Description 

Figure  1  details  the  SIT  LSP  communications  network.  The  involved  facilities  at  Point  Mugu,  CA 
and  China  Lake,  CA  were  already  linked  together  through  the  Naval  Air  Warfare  Center  Weapons 
Division  (NAWCWPNS)  Real-Time  Network  (NRNet)  at  Point  Mugu.  JADS  JTF  leased  a  T1 
line  from  Albuquerque,  NM  to  Point  Mugu,  CA  and  connected  into  the  NRNet  at  the  Sea  Range 
Communications  Center.  Because  of  latency  concerns  and  NRNet  traffic  loading,  the  missile 
simulator  and  F/A  18  Labs  at  China  Lake  had  to  connect  their  MIL  STD  1553B  Data  Busses  via  a 
separate  data  circuit  in  order  to  get  proper  interaction  between  the  F/A  18  weapons  control 
system  and  the  missile  launch  control  system.  Refer  to  Reference  1  for  detailed  network  diagrams 
and  test  description. 


KG- 194  CSU/DSU  CSU/DSU 


Land  Range 
Comm  Center 
China  Lake,  CA 


CSU/DSU  CSU/DSU  KG-194 


Figure  1  SIT  Linked  Simulator  Phase  Network 

2.2.3  Systems  Integration  Test  (LSP) 

Test  control  procedures  were  designed  for  simplicity  and  decentralization  of  control.  The  design 
of  the  test  required  the  shooting  aircraft  to  achieve  a  certain  geometry  where  altitude,  airspeed, 
range  to  target,  target  aspect,  missile  pointing  angle  and  target  acceleration  were  simultaneously 
achieved.  The  only  way  these  conditions  could  be  repeated  consistently  was  if  the  shooter  pilot 
controlled  the  entire  intercept.  To  reduce  the  number  of  variables  he  had  to  deal  with,  the  shooter 
pilot  was  “unfrozen”  at  a  specified  distance  aft  of  the  target  with  approximately  a  half  mile  of 
lateral  separation.  After  both  simulators  were  reset  and  frozen,  the  target  simulation  was  started. 
The  shooter  watched  his  HUD  as  the  target  range  opened.  When  he  saw  the  range  to  target  he 
wanted,  he  unfroze  his  own  simulation.  This  procedure  was  used  instead  of  simultaneously 
unfreezing  both  simulations  since  the  time  required  to  unfreeze  the  target  simulation  took  from 
less  than  a  second  to  up  to  3  seconds.  This  procedure  permitted  the  shooter  to  achieve  a 
consistent  starting  geometry  without  having  to  make  a  difficult  and  time  consuming  speed 
adjustment  to  achieve  proper  range.  The  initial  altitude  difference  and  lateral  separation  were 
determined  by  trial  and  error. 

When  the  test  controller  saw  both  aircraft  “flying,”  he  called  for  the  target  to  turn  and  cross  in 
front  of  the  shooter  (see  Fig.3).  The  target  then  flew  a  turn  at  constant  speed,  altitude  and  turn 
acceleration.  This  flight  path,  combined  with  the  initial  conditions  of  each  simulator,  allowed  the 
shooter  to  achieve  the  desired  range  and  aspect  parameters  with  little  maneuvering. 

Once  the  test  controller  had  started  the  target  turning,  achieving  the  desired  geometry  was  totally 
under  the  control  of  the  F/A-18  pilot.  After  the  missile  finished  its  flyout  the  test  controller 
terminated  the  run  and  released  the  simulators  to  reset  following  any  quick  look  procedures  they 
needed  to  perform. 

All  analysts  were  located  in  the  SIMLAB  where  the  data  from  the  missile  were  presented.  The 
only  communication  required  from  the  SIMLAB  was  a  statement  on  whether  the  run  was 
acceptable  or  not.  This  control  approach  resulted  in  minimal  communications  on  the  unclassified 
command  voice  circuit.  For  a  more  detailed  explanation  and  description  of  the  LSP  test  effort 
refer  to  Reference  1  (SIT,  LSP  Final  Report). 
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Figure  3  (Pre-Launch  Shooter  and  Target  Trajectories  (Run  #9  on  11/19/96)  -  “God’s- 

Eye”  View) 

2.2.4  Test  Monitoring  Displays 

Two  separate  displays  were  developed  to  permit  effective  control  of  the  tests  and  to  evaluate  how 
well  each  intercept  was  performed. 

-  The  first  display  was  a  two-dimensional  (“God’s-eye”)  stealth  viewer.  Entity  state  PDUs 
from  the  two  aircraft  and  the  missile,  after  it  was  fired,  were  displayed  on  a  screen 
showing  an  outline  of  the  China  Lake  restricted  airspace.  Since  both  players  were  virtual, 
the  exact  location  did  not  matter.  Each  entity  was  represented  by  a  friendly  or  hostile 
symbol  with  an  aspect  vector  whose  direction  represented  the  entity  heading  and  whose 
length  was  proportional  to  the  speed.  Each  entity  created  a  “trail”  of  points  representing 
its  flight  path.  Because  the  LSP  was  attempting  to  replicate  a  single  profile,  the  ground 
track  for  each  run  essentially  overlaid  the  tracks  of  the  previous  runs.  This  provided  a 
useful  capability  for  making  a  first  cut  evaluation  of  how  well  the  intercept  and  flyout 
replicated  the  desired  trajectories. 

-  The  other  “discretes”  display  showed  various  parameters  which  defined  the  “shot  box.” 
Target  altitude,  mach  number,  and  turning  acceleration,  long  with  shooter  altitude  and 
mach  number  were  displayed  in  strip  chart  format  with  a  total  of  16  seconds  of  history 
trace  visible  at  a  time.  A  horizontal  blue  line  was  displayed  continuously  for  the  desired 
value  for  each  parameter  throughout  the  period  when  the  instantaneous  value  was  being 
displayed.  A  digital  readout  for  each  parameter  was  also  provided.  When  each  parameter 
fell  within  the  desired  range  of  values  which  defined  the  shot  box,  the  digital  readout 
turned  from  white  to  green. 
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The  speed  with  which  the  shooter  was  closing  on  the  target  (Vc)  was  also  displayed  in  this 
manner;  however,  this  parameter  turned  out  not  to  be  useful. 

The  radar  mode  and  status  of  master  arm  power  and  the  trigger  were  listed  in  gray  and  turned 
green  when  selected  or  the  trigger  was  squeezed. 

Range  to  target  was  shown  to  one  decimal  place  of  precision.  The  numbers  were  green  when 
within  the  desired  range. 

The  true  heading  of  each  aircraft  was  displayed  in  digital  format. 

Two  circular  displays  showed  (a)  where  the  radar  was  pointing  relative  to  the  nose  of  the  aircraft 
both  in  azimuth  and  elevation  and  (b)  the  angle  between  the  nose  of  the  target  and  the  line-of- 
sight  to  the  shooter  (i.e.,  the  target  aspect  angle). 

The  radar  pointing  angle  was  used  as  a  reasonably  close  measure  of  how  far  off  boresight  the 
missile  seeker  was  pointing  since  the  seeker  was  slaved  to  the  radar  line-of-sight  at  launch. 

Each  display  turned  from  white  to  green  when  the  particular  parameter  met  acceptable  launch 
conditions.  This  provided  a  very  simple  and  accurate  method  of  determining  whether  the 
particular  run  met  the  desired  parameters.  However,  the  final  decision  on  whether  to  count  the 
run  as  “in  the  shot  box”  rested  with  the  analysts  in  the  SIMLAB. 

2.3  Live  Fly  Phase  (LFP)  Test  Control  Requirements 

1.  Unclassified  conference  call  network  to  support  up  to  10  phones  for: 

a.  Ordering  start/stop  of  data  loggers 

b.  Communication  between  test  controller  in  TCAC  and  Test 
Director  located  at  the  CCF. 

2.  Classified  STU-3  phone  capability  between  three  sites  (CCF,  MISILAB 

and  TCAC  for  discussion  of  classified  issues. 

3.  Internal  communications  at  the  CCF  between  JADS  Test  director  and  the  aircraft 
controller. 

4.  2-D  display  of  target/shooter/missile  engagement  in  the  TCAC. 

5.  Quick-look  analysis  capability  to  verify  proper  execution  of  last  engagement. 


2.3.1  LFP  Network  Description 

Figure  3  details  the  SIT  LFP  communications  network.  The  involved  facilities  at  Eglin  AFB,  FL 
built  a  network  infrastructure  in  order  to  meet  the  SIT  LFP  requirements.  JADS  JTF  leased  a  T1 
Line  from  Albuquerque,  NM  to  Eglin  AFB,  FL  and  connected  into  Eglin’ s  network  at  the  Central 
Control  Facility.  The  network  required  minor  changes  once  all  of  the  components  were  installed 
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in  order  to  optimize  network  performance  and  meet  the  LFP  network  requirements.  Refer  to 
Reference  2  for  detailed  network  diagrams  and  test  description. 


Missile  Lab 
Eglin  AFB,  FL 


Cisco  7010 
Router 


x  Fiber-Optic 
X  MODEM 


Open  Air  Range^^^A 
Eglin  AFB,  FL 


F-16 

Target 


idnx-20  Igggl- 

Multiplexor 


KTV-7HS 


JADS  JTF  TCAC 
Albuquerque,  NM 


KIV-7HS  IDNX-20 
Multiplexor 


Figure  3  SIT  Live  Fly  Phase  Network 


2.3.2  Control  Procedures 

Test  control  was  centralized  at  the  CCF,  rather  than  the  TCAC,  due  to  the  following  factors: 

-  The  live  aircraft  had  to  be  controlled  by  air  traffic  controllers  from  the  CCF  (due  to  range 
policy  which  was  driven  primarily  by  range  safety  issues). 

-  During  the  LSP,  test  control  was  centralized  in  the  TCAC,  and  the  test  controller  in 
the  TCAC  provided  positive  control  of  the  pilots  “flying”  the  aircraft  simulators.  This 
control  was  made  possible  by  real-time  data  and  position  displays  in  the  TCAC.  Also, 
there  were  no  flight  safety  concerns  since  only  simulators  were  being  used. 

—  Air  traffic  controllers  could  not  exercise  safe  control  of  the  aircraft  from  the  TCAC. 
Entity  state  data  used  to  drive  the  aircraft  position  displays  in  the  TCAC  had  too  much 
latency  for  safe  aircraft  control  (nearly  2.5  seconds  of  latency,  as  Table  4.3.4.3-1 
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shows).  Further,  auxiliary  data  on  range  weather  conditions  and  other  live  aircraft  in 
the  test  area  were  not  available  at  the  TCAC. 

-  Communications  with  the  key  participants  was  most  effectively  exercised  from  the 
CCF.  The  JADS  test  director,  the  test  conductor,  and  the  CCF  coordinator  (see  next 
paragraph)  had  direct  physical  contact  at  the  CCF  and  communications  links  to  the 
other  nodes. 

Test  control  centered  on  three  individuals  located  at  the  CCF: 

-  JADS  test  director  (SIT  Team  Lead):  Determined  next  test  profile  and  informed  test 
conductor.  Made  decisions  on  any  changes  to  planned  profiles  and  on  when  to  terminate 
the  mission.  All  nodes  within  the  LFP  architecture  had  on-scene  JADS  representatives 
with  the  ability  to  speak  to  the  team  lead  in  the  CCF.  This  provided  the  team  lead  with 
the  potential  capability  to  react  and  affect  the  conduct  of  the  mission  in  a  real-time 
manner. 

Test  Conductor:  Announced  the  start  and  stop  of  each  pass,  the  flight  profile  to  be 
executed  (i.e.,  test  card  number),  and  the  pass  number  to  the  CCF  control  net  and  to  the 
air  traffic  controllers.  The  air  traffic  controllers  relayed  this  information  to  the  pilots  and 
vectored  the  aircraft  to  starting  positions  for  each  pass. 

-  CCF  Coordinator:  Relayed  the  “30  seconds  until  pickle”  warning  from  the  test  conductor 
to  the  MISILAB.  Polled  players  on  results  of  each  pass. 

For  a  more  detailed  explanation  and  description  of  the  LFP  test  effort  refer  to  Reference  2  (SIT, 
LFP  Final  Report). 

2.4  End-to-End  (ETE)  Test  Control  Requirements 

The  ETE  test  control  had  to  support  the  following: 

1.  Unclassified  conference  call  network  to  support  up  to  10  phones  for: 

a.  Ordering  start/stop  of  data  loggers. 

b.  Communication  between  the  Northrop  Grumman  SATCOM  operator  and  the 
TCAC. 

c.  Communication  between  test  controller  in  TCAC  and  test  director  located  at 
FT  Hood. 

2.  Classified  T-l  phone  capability  between  three  sites  (Northrop  Grumman,  FT  Hood 
and  TCAC  for  discussion  of  classified  issues. 

3.  Unclassified  T-l  phone  capability  between  four  sites  (FT  Hood,  FT  Sill,  TRAC 
WSMR  and  the  TCAC 

4.  HF  and  UHF  radio  link  between  the  test  director  at  FT  Hood  and  the  JADS  team 
member  onboard  the  E-8C. 


2.4.1  End-to-End  Test  Network  Description 
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Figure  4  details  the  ETE  communications  network.  The  ETE  test  also  presented  the  challenge  of 
transmitting  data  from  an  unclassified  site  into  a  classified  network.  This  was  accomplished  by 
configuring  a  one  way  only  link  at  the  JADS  facility.  JADS  JTF  received  unclassified  (non-secure) 
data  from  White  Sands  Missile  Range  (WSMR),  NM,  and  Ft.  Sill,  OK  and  forwarded  that  data 
into  the  TCAC,  and  prevented  any  data  from  the  secure  network  propagating  to  the  non-secure 
network..  Refer  to  Reference  3  for  detailed  network  diagrams  and  test  description. 


Northrup  Grumman  Ft-  Hood,  TX  Ft.  Hood,  TX  FT.  Sill,  OK  WSMR,  NM 

Melbourne,  FL  LGSM 

Figure  4  ETE  Test  Network 
2.4.2  Test  Control  and  Monitoring 

During  Phase  4,  ETE  Test  and  Network  and  Engineering  team  members  performed  test  control 
from  the  TCAC.  Test  control  consisted  of  three  major  areas  -  network  monitoring, 
communications,  and  test  procedures. 

The  Network  and  Engineering  team  conducted  network  monitoring  in  the  TCAC  using  hardware 
and  software  tools.  The  software  consisted  of  commercial  products  and  test-specific  tools 
developed  by  JADS  analyst/programmers.  They  used  the  following  systems. 

Silicon  Graphics,  Inc.  (SGI)  Indy  -  JADS  logger 
SGI  Indy  -  time  server 
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SGI  Indigo  -  NETVisualyzer™ 

SUN  SPARC  5  -  Spectrum™ 

Line  printers 

JADS  analyst/programmers  developed  the  JADS  logger.  This  software  recorded  all  PDU  traffic 
at  individual  sites  as  well  as  displaying  the  PDU  rate  in  real  time.  All  nodes,  with  the  exception  of 
Fort  Hood,  had  a  JADS  logger  installed.  All  traffic  into  and  out  of  Ft  Hood  was  transmitted  using 
tactical  protocols  and  formats  rather  than  DIS.  Each  tactical  system  had  some  level  of  logging  of 
internal  operations,  however,  these  databases  or  reports  proved  to  be  of  minimal  use  to  the  test 
team.  The  JADS  logger  recorded  the  receipt  of  the  PDU  and  time  stamped  it  using  an  accurate 
time  source.  These  data  were  used  to  analyze  PDU  transmission  performance  over  the  network. 

JADS  analyst/programmers  also  developed  a  time  server  which  provided  the  accurate  time  source 
needed  for  the  JADS  logger.  The  time  server  was  tied  to  a  global  positioning  system  (GPS) 
receiver  located  in  the  TCAC  and  provided  time  to  all  instrumentation  nodes  with  an  accuracy  of 
100  microseconds.  The  software  also  contained  monitoring  tools  to  track  the  time  servers 
performance  over  an  8-hour  test  period. 

Cabletron  Spectrum™,  NETVisualizer™,  and  line  printers  were  used  to  provide  network 
monitoring.  Spectrum™  measured  bandwidth  utilization.  This  tool  recorded  the  percentage  of 
bandwidth  used,  as  well  as  bandwidth  loading  on  a  network  segment.  NETVisualizer™  software 
displayed  real-time  bandwidth  use  in  a  rolling  bar  graph  format  for  quick  visual  reference.  The 
line  printers  provided  a  printout  of  network  router  status.  Any  failure  or  high  bit  error  rate 
resulted  in  a  printout  showing  the  problem  and  identity  of  the  offending  router. 

Communication  among  the  distributed  ETE  Test  nodes  was  critical.  The  TCAC  provided  all 
needed  communication  systems.  Dedicated  phone  circuits,  residing  on  the  T-l  lines,  provided 
classified  and  unclassified  service.  The  unclassified  line  allowed  connectivity  among  the  TCAC, 
Fort  Hood,  Fort  Sill,  and  TRAC-WSMR.  The  classified  line  allowed  connectivity  among  the 
TCAC,  Fort  Hood,  and  Northrop  Grumman.  These  lines  allowed  the  operator  to  select  the 
desired  site,  lift  the  receiver,  and  connect  directly  to  that  site.  The  TCAC  could  select  multiple 
sites  for  conference  calls  on  these  lines.  In  addition  to  these  lines,  the  ETE  Test  team  used  an 
unclassified  conference  line  to  coordinate  such  test  events  as  network  checks  and  after-action 
debriefs.  This  line  allowed  up  to  ten  participants  to  connect  at  one  time. 

There  were  additional  requirements  to  communicate  between  the  E-8C  and  the  test  director  on 
the  ground  at  FT  Hood.  Secure  HF  and  UHF  links  were  established  between  the  JADS  test  team 
member  onboard  the  aircraft  when  the  aircraft  arrived  on  station  over  FT  Hood.  The  JADS 
member  communicated  the  status  of  the  test  as  well  as  any  problems  to  the  test  director.  This 
communication  link  was  also  used  by  the  JADS  analyst  to  coordinate  data  collection  onboard  the 
E-8C  and  the  LGSM  located  at  FT  Hood.  Communications  between  the  E-8C  and  Northrop 
Grumman  SATCOM  operators  located  at  Melbourne,  Florida  was  accomplished  via  a  HF  link  or 
by  using  text  message  traffic  over  the  SATCOM  link.  These  links  were  used  to  communicate  the 
status  of  the  SATCOM  link  and  to  coordinate  the  start  of  the  scenario  with  the  test  controller  in 
the  TCAC. 
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Test  procedures  were  required  to  provide  effective  control  of  all  test  nodes  during  test  events. 
The  test  procedures  were  in  checklist  format,  which  provided  for  standardization  among  the 
distributed  nodes.  The  network  checklist  was  most  critical  and  was  used  to  initialize  the  network 
before  the  test.  Other  checklists  included  those  used  to  start  up  hardware  and  software  at 
individual  nodes,  as  well  as  the  checklist  used  by  the  TCAC  test  controller  to  start  and  stop  the 
overall  test. 

For  a  more  detailed  description  and  explanation  of  the  ETE  test  effort  refer  to  Reference  3  (ETE 
Test  Interim  Report  Phase  4). 

2.5  Electronic  Warfare  (EW)  Test  Control  Requirements 

The  EW  test  control  had  to  support  the  following: 

1.  Unclassified  conference  call  network  to  support  up  to  10  phones  for: 

a.  Test  briefs  with  all  participants. 

b.  Offline  communications  between  multiple  sites 

2.  Classified  phone  capability  between  three  sites  (AFEWES,  ACETEF  and  TCAC)  for 
test  control  and  discussion  of  classified  issues. 

3.  Visual  display  of  engagement  for  test  director/controller. 

4.  Quick-look  analysis  capability  to  verify  proper  execution  of  last  engagement. 

5.  Visual  display  of  Federate  health  status 


2.5.1  Electronic  Warfare  Test 

Figure  5  details  the  EW  communications  network.  The  EW  test  presented  many  challenges  in  the 
networking  equipment’s  configuration.  Although  the  EW  test  was  able  to  utilize  the  same 
equipment  as  SIT  and  ETE,  the  network  equipment  required  extensive  configuration  changes  in 
order  to  optimize  the  routers’  performance  while  reducing  transmission  latency  to  minimal  levels. 
Refer  to  Reference  4  for  detailed  network  diagrams  and  test  description. 
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AFEWES  LAB 
Ft.  Worth,  TX 


ACETEF 

Patuxent  River,  MD 


Figure  5  EW  Test  Network 


2.5.2  Test  Control 

Test  control  is  an  important  aspect  of  any  formal  test  environment.  To  insure  that  test  conditions 
and  status  were  monitored  and  maintained  in  a  distributed,  ADS-based  test  environment,  JADS 
had  to  carefully  define  its  requirements.  Areas  addressed  included  test  control,  unique  real-time 
status  reporting  and  display  capabilities,  and  test  control  procedures  within  the  TCAC  as  well  as 
AFEWES  and  ACETEF.  Critical  components  of  the  Phase  3  test  control  included  federation  time 
synchronization,  federation  startup,  and  status  monitoring  described  below.  For  a  more  detailed 
description  and  explanation  of  the  EW  test  effort  refer  to  Reference  4  (EW  Test  Interim  Report 
Phase  3). 

2.5.3  Federation  Time  Synchronization 

All  computers  used  for  the  Phase  3  test  used  a  common  time  source  to  insure  valid  time  values 
were  recorded  in  the  logs  at  all  sites  during  test  execution.  Prior  to  the  start  of  daily  testing,  a 
time  synchronization  test  was  run  by  the  TCAC  to  verify  that  all  computers  were  using  the 
correct,  accurate  time  source  (IRIG-B)  in  their  operations  and  not  running  on  local  internal 
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system  time.  All  JADS  federates  except  AFEWES  and  the  Analysis  Federate  participated  in  the 
time  synchronization  test.  Since  the  Analysis  Federate  only  read  and  recorded  data,  it  recorded 
time  synchronized  data  from  other  federates.  Time  synchronization  was  performed  by  initiating  a 
normal  run  and  after  a  jamming  response  was  observed  in  the  TCAC  the  execution  was  stopped. 
JADS  copied  the  TCF  log  file  to  the  appropriate  log  file  directory  and  ran  the  logfile_summary 
program  on  the  file.  The  recorded  times  for  all  messages  in  federate  log  files  were  checked  to 
look  “reasonable”  (within  +/-  20  milliseconds  of  each  other).  The  times  recorded  on  the  Link 
Health  Check  messages  showed  the  time  on  all  of  the  SGI  02s.  Execution  control  messages 
(Start  and  Stop)  showed  the  time  for  the  ADRS  2  PC.  The  time  recorded  on  the  X_File_Spec 
message  showed  the  time  for  the  ADRS  1  PC.  The  time  recorded  on  the  SUT  messages  showed 
the  time  for  the  Jammer  federate. 

The  AFEWES  computers  were  also  synchronized  to  an  IRIG-B  time  source.  Software  executed 
on  the  AFEWES  computers  synchronized  the  system  time  to  the  IRIG-B  time.  The  AFEWES 
federate  software  used  the  system  time  as  its  time  source.  To  determine  if  AFEWES  is 
synchronized,  JADS  would  run  the  raw  TCP  test  program  previously  used  for  RTI  testing.  Upon 
completion,  the  receiver  printed  out  the  minimum,  maximum,  and  mean  latency.  JADS  verified 
that  the  latency  was  “reasonable”.  A  real-time  analysis  capability  was  not  available  to  determine  if 
all  systems  maintained  time  synchronization  during  test  runs.  However,  during  subsequent  data 
analysis  the  quality  of  time  synchronization  was  able  to  be  determined. 

2.5.4  Federation  Startup 

During  the  previous  integration  tests  for  the  Phase  2  test,  it  was  determined  there  were  less  data 
dropouts  if  the  Platform  federate  was  not  the  first  federate  to  join  the  federation.  Since  five 
federates  in  the  TCAC  were  all  subscribing  to  most  of  the  same  reliable  data,  DMSO  technical 
support  recommended  that  JADS  execute  with  a  single  reliable  distributor  for  the  TCAC. 
Normally,  each  federate  contained  a  reliable  distributor  to  send  and  receive  reliable  data.  There 
was  a  TCP  connection  from  every  reliable  distributor  to  every  other  one.  By  configuring  the 
TCAC  federates  to  use  one  reliable  distributor,  the  number  of  TCP  connections  and  the 
subsequent  traffic  on  the  WAN  was  reduced.  The  RFENV  federate  did  not  have  much  data  to 
process,  so  it  was  chosen  as  the  host  location  for  the  TCAC  reliable  distributor  (RELDISTR). 

The  federate  containing  the  reliable  distributor  started  the  federation  executive  (FEDEX)  and 
joined  the  federation  first.  It  was  observed  that  if  the  FEDEX  was  not  started  on  the  same 
computer  where  the  RTI  executive  (RTIEXEC)  was  running,  it  would  take  a  few  minutes  for  the 
FEDEX  to  find  the  RTIEXEC. 

After  the  RFENV  federate  started  and  joined  the  federation,  all  of  the  other  federates  in  the 
TCAC  would  join  (staggered  by  a  second  or  two).  After  the  TCAC  federates  successfully  joined 
the  federation,  the  remote  federates  (AFEWES  and  Jammer)  joined. 

2.5.5  Federate  Status  Monitoring 

The  Link  Health  Check  display  as  well  as  the  FEDEX  window  were  monitored  during  federation 
execution.  If  there  were  problems,  generally  one  of  these  windows  displayed  an  indication.  For 
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example,  the  Link  Health  Check  status  display  on  the  TCF  screen  indicated  federate  status  as  red 
or  green.  It  provided  the  capability  to  monitor  specific  multicast  traffic  paths  in  one  direction  only 
between  any  two  federation  nodes  (e.g.,  JADS  -  AFEWES).  The  FEDEX  window  monitored  the 
RTFs  “heartbeat”  messages  over  the  reliable  (TCP)  data  paths.  The  federate  displayed  red  if  the 
federate  stopped  sending  Link  Health  messages  due  to  software  error  or  a  failure  in  the  link. 
However,  the  problem  was  generally  noted  by  one  of  the  remote  sites  before  the  TCAC  federates 
exhibited  a  corresponding  display.  This  was  due  to  the  fact  that  the  TCAC  had  fewer  indications 
of  information  outages  from  other  nodes.  The  link  health  and  FEDEX  windows  could  only  detect 
outages  that  occurred  over  an  extended  period  of  time  (>  3  seconds  for  link  health,  20  seconds  for 
FEDEX).  Status  monitoring  procedures  of  individual  federates  and  operator  procedures  are 
described  below  in  further  detail. 

2.5.6  Jammer  (ACETEF)  Operation 

For  phase  3,  the  facilities  at  ACETEF  hosted  the  ALQ-131  jammer.  The  ACETEF  Jammer 
federate  operation  involved  two  major  activities  for  each  test  run. 

The  first  activity  managed  the  local  processes  to  successfully  join  the  federation  and  begin 
accepting  data.  This  complex  procedure  consists  of  several  steps.  The  first  step  was  to  run  a 
script  which  started  the  Link  Health  Check  Monitor,  the  Emitter  Monitor,  a  graphic  visualizer 
called  “TacPlot”,  SWEG,  and  the  RTI  and  SWEG  interfaces,  which  comprised  the  ACETEF 
HLA  interface.  Immediately,  SWEG  would  begin  polling  for  “external  assets”  and  continued  to 
do  so  until  all  external  assets  were  “ready”.  The  external  assets  consisted  of:  the  emitters  being 
controlled  by  ATEWES,  the  HLA  interface,  SWEG  or  TacPlot  graphics,  and  the  asset  to  asset 
communications.  The  RTI  interface  immediately  polled  the  user  for  the  run  number,  set  up  a 
shared  memory  connection  with  the  SWEG  interface,  started  the  JADS  logger  file,  joined  the 
FEDEX,  published  and  subscribed  to  all  expected  data,  and  finally  rested  in  a  pattern  of  polling 
the  RTI  for  a  start  command  (issued  by  TCAC)  and  broadcasting  the  status  of  the  federate.  The 
SWEG  interface  meanwhile,  established  its  shared  memory  connection  to  the  RTI  interface  and 
waited  for  the  start  command  issued  by  the  RTI  interface. 

The  ACETEF  federate  operator  then  vocally  authorized  the  ATEWES  operator  to  start,  and  once 
ATEWES  came  on  line,  SWEG  recognized  this  and  passed  to  its  next  waiting  step.  SWEG  status 
was  then  waiting  for  a  “start  execution”  order  from  the  SWEG  Interface. 

The  ACETEF  federate  operator  informed  the  JADS  TCAC  that  ACETEF  was  ready  for  the  start. 
Another  good  visual  queue  that  ACETEF  was  ready  for  the  start  command  was  when  the  SWEG 
graphics  screen  was  displayed. 

JADS  then  sent  the  start  command  from  the  TCAC,  which  was  received  at  the  RTI  Interface  and 
passed  to  the  SWEG  Interface,  which,  in  turn,  then  sent  SWEG  the  “start  execution”  command. 
At  this  point,  SWEG  graphics  would  appear  and  the  federate,  and  the  federation,  entered  “run” 
mode. 


17 


The  operator  monitored  the  integrity  of  the  federation  with  the  tools  available  to  him  at  ACETEF. 
This  included  the  Link  Health  Check  Monitor,  the  Emitter  Monitor,  SWEG  or  TacPlot  graphics, 
and  warning  or  dropout  message  information  displayed  on  screen  by  the  RTI  Interface.  The  Link 
Health  Check  monitor  indicated  federate  status  and  link  check  between  federation  nodes  by  red  or 
green  coloring.  The  emitter  monitor  displayed  the  different  emitters  used  in  the  scenario  and  their 
respective  modes  of  operation  were  indicated  by  red  or  green  coloring.  Warning  or  drop  out 
messages  were  displayed  to  indicate  missing  and  out  of  sequence  data.  If  necessary,  dead 
reckoning  smoothed  lost  TSPI  data.  SWEG  graphics  or  TacPlot  were  used  for  test  execution 
control. 

At  the  end  of  the  run,  the  ACETEF  federate  operator  shut  down  the  Link  Health  Check  and 
Emitter  Monitors,  the  RTI  and  SWEG  Interfaces,  TacPlot,  and  SWEG.  The  ATEWES  operator 
shut  down  the  ATEWES. 

At  the  end  of  each  test  day,  the  ACETEF  federate  operator  collected  all  log  files  and  organized 
them  into  permanent  directories.  The  ACETEF’s  federate  logger  file,  the  ATEWES  timing  file 
and  the  ATEWES  dx  file  were  all  collected.  These  files  consumed  many  megabytes  of  storage. 

2.5.7  Test  Control  Federate  (TCF)  Operation 

The  TCF,  located  in  the  TCAC,  started  by  executing  a  shell  script.  When  the  federate  started,  it 
prompted  the  operator  for  a  run  number.  After  the  operator  entered  a  run  number,  the  federate 
prompted  the  operator  to  join  the  federation.  When  directed  by  the  JADS  test  controller,  the 
operator  entered  the  federation.  When  the  TCF  completely  joined  the  federation,  the  ADRS 
software  was  started  by  the  ADRS  operators  on  each  of  the  three  ADRS  PCs.  The  software  on 
each  PC  started  a  few  seconds  apart  from  each  other  to  help  prevent  the  TCF  federate  from 
periodically  crashing.  Once  the  ADRS  software  was  running,  the  federate  waited  for 
Execution_Control  commands  from  ADRS.  When  an  execution  control  attribute  was  received, 
the  TCF  federate  published  it  for  the  other  federates.  The  script  names  and  the  start  and  stop 
messages  were  sent  to  all  of  the  federates  using  this  method.  When  the  command  was  an 
Execution_Control  attribute  with  an  Execution_Control_Word  that  indicated  stop  test  execution, 
the  TCF  federate  published  the  attribute  and  then  resigned  from  the  federation. 

2.5.8  Platform  Federate  Operation 

The  Platform  federate  located  in  the  TCAC  started  by  executing  a  shell  script.  When  the  federate 
started,  it  prompted  the  operator  for  a  run  number.  After  the  operator  entered  a  run  number,  the 
federate  prompted  the  operator  to  join  the  federation.  When  directed  by  the  JADS  test 
controller,  the  operator  entered  the  federation.  The  Platform  federate  waited  until  it  received  an 
attribute  update  that  contained  the  name  of  the  script  to  be  loaded.  When  it  received  the  script 
name ,  the  federate  displayed  the  name  of  the  script  being  loaded.  The  federate  then  waited  for  an 
execution  control  attribute  update  with  that  indicated  the  start  of  test  execution. 

During  a  federation  execution,  the  Platform  federate  executed  without  operator  intervention.  The 
window  in  which  the  federate  executed  was  monitored  for  error  messages.  The  Platform  federate 
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played  its  script  until  an  execution  control  attribute  update  was  received  that  indicated  stop  of  test 
execution.  The  federate  then  resigned  from  the  federation. 

2.5.9  Radio  Frequency  Environment  (RFENV)  Federate  Operation 

The  RFENV  federate  located  in  the  TCAC  started  first  because  it  created  the  federation  execution 
(FEDEX)  and  the  reliable  distributor  (reldistr)  used  by  all  federates  in  the  TCAC.  The  RFENV 
federate  started  by  executing  the  a  shell  script.  When  the  federate  started,  it  prompted  the 
operator  for  a  run  number.  After  the  operator  entered  a  run  number,  the  RFENV  federate  created 
the  FEDEX.  The  federate  then  prompted  the  operator  to  join  the  federation.  The  operator 
entered  a  ‘y’  at  the  prompt  so  that  the  federate  joined  the  federation.  The  RFENV  federate 
waited  until  it  received  an  attribute  update  that  contained  the  name  of  the  script  to  be  loaded. 
When  it  received  the  script  name ,  the  federate  displayed  the  name  of  the  script  being  loaded. 
Upon  completion  of  script  loading,  the  federate  displayed  “done”.  The  federate  waited  for  an 
“execution  control  attribute”  update  that  indicated  start  of  test  execution. 

During  a  federation  execution,  the  RFENV  federate  ran  without  operator  intervention.  The 
operator  monitored  the  window  in  which  the  federate  executed  for  error  messages.  The  FEDEX 
window  was  also  monitored  for  error  messages  indicating  loss  of  contact  with  other  federates. 

The  RFENV  played  its  script  until  an  execution  control  attribute  update  was  received  that 
indicated  the  stop  of  test  execution.  The  RFENV  federate  waited  for  all  other  federates  to  resign 
from  the  federation  and  then  resigned  from  and  destroyed  the  federation  execution.  If  there  were 
problems  with  any  federate  resigning,  the  federation  was  destroyed  manually  by  entering  a  “kill” 
command  in  the  FEDEX  window. 

2.5.10  Terminal  Threat  Hand-Off  (TTH)  Federate  Operation 

The  TTH  federate  located  in  the  TCAC  started  by  executing  a  shell  script.  When  the  federate 
started,  it  prompted  the  operator  for  a  run  number.  After  the  operator  entered  a  run  number,  the 
federate  prompted  the  operator  to  join  the  federation.  When  directed  by  the  JADS  test  controller, 
the  operator  entered  the  federation.  The  TTH  federate  waited  until  it  received  an  attribute  update 
containing  the  name  of  the  script  to  be  loaded.  When  it  received  the  script  name,  the  federate 
displayed  the  name  of  the  script  being  loaded.  Upon  completion  of  script  loading,  the  federate 
displayed  “done”.  The  federate  then  waited  for  an  execution  control  attribute  update  with  an 
Execution_Control_Word  indicating  the  start  of  test  execution. 

During  a  federation  execution,  the  TTH  federate  executed  without  operator  intervention.  The 
window  in  which  the  federate  executed  was  monitored  for  error  messages.  The  federate  played 
its  script  until  an  execution  control  attribute  update  was  received  that  indicated  the  stop  of  test 
execution.  The  TTH  federate  then  resigned  from  the  federation. 

2.5.11  AFEWES  Threats  Federate  Operation 

The  AFEWES  threats  federate  consisted  of  federate  software  hosted  on  an  SGI  computer,  and 
facility  unique  systems  and  software  for  scenario  status  control  and  display,  test  management 
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centers,  and  operator  consoles.  AFEWES  controlled  the  test  run  execution  and  individual 
systems  from  a  central  facility  linked  internally  by  intercoms  with  external  voice  links  to  JADS  and 
ACETEF.  The  test  controller  at  JADS  advised  the  AFEWES  controller  by  voice  for  federate  and 
run  start  and  stop  conditions  similar  to  the  ACETEF  federate.  The  AFEWES  controller  then 
coordinated  internal  execution  actions  with  operators  and  advised  JADS  of  current  status. 

2.5.12  Analysis  Federate  Operation 

The  Analysis  federate  provided  an  improved  scenario  viewer  for  observing  north  and  south  bound 
test  runs  and  specific  threat  engagements.  It  showed  the  specific  modes  a  threat  site  used,  and 
missile  flyouts  as  they  occurred.  Real-time  displays  of  the  10  EW  MOPs,  and  the  real-time  values 
of  jamming-to-signal  ratio  and  tracking  error  were  provided  for  situational  awareness  of  each 
threat  engagement  with  the  target  aircraft.  Data  collection  and  storage  for  the  Analysis  federate 
were  nearly  automatic,  and  required  little  operator  intervention  once  the  run  started.  Once  the 
federation  began,  the  Analysis  Federate  operator  waited  to  join  the  federation.  Queued  from  the 
TCF  operator,  the  Analysis  Federate  joined  the  federation  and  awaited  the  start  of  the  test 
execution.  During  the  run,  the  window  was  monitored  for  errors  in  the  threat  performance  or 
variance  in  the  threat  operators.  Once  the  test  execution  completed,  the  operator  resigned  the 
Analysis  federate  from  the  federation  and  immediately  restarted  the  software  to  begin  the  next 
run. 


3.0  Equipment  Descriptions 


3.1  Voice  Communications 

3.1.1  Voice  Networks 

The  JADS  tests  had  a  requirement  that  all  participants  be  able  to  communicate  in  a  conference 
call.  Two  alternatives  were  used  to  satisfy  the  conferencing  requirement.  Existing  unclassified 
conference  bridges  were  used  for  the  SIT  and  ETE  tests.  The  conference  bridges  were  provided 
by  the  base/post  communications  center  or  by  one  of  the  test  facilities.  All  personnel  would  call 
into  the  conference  bridge  at  specified  times.  A  means  of  calling  an  unscheduled  conference  had 
to  be  determined.  This  involved  using  standard  point-to-point  calls,  beepers,  and  computer  talk 
sessions. 

3.1.1.1  Conference  Bridges 

The  SIT  required  multiple  conference  calls:  one  for  the  pilots  and  controllers  and  one  for 
the  test  controller,  site  observers,  and  data  collection  personnel.  For  the  SIT  LSP,  the 
conference  bridge  was  supplied  by  NAWCWPNS.  For  the  SIT  LFP  the  test  control  voice 
network  was  used  for  pilots  and  aircraft  control.  An  unclassified  conference  bridge 
capability  provided  by  the  Kirtland  AFB  communications  squadron  was  used  for  site 
observers  and  data  collectors. 
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For  the  ETE  test,  a  single  conference  call  was  used  by  all  participants.  Unlike  the  SIT  and 
EW  tests,  the  conference  calls  were  not  held  throughout  the  duration  of  the  test.  Since  a 
test  period  was  typically  7  hours  in  duration  with  little  interaction,  this  would  have  been 
impractical.  Conferences  were  held  at  predetermined  times  before  a  test  in  order  to  brief 
all  participants  and  startup  the  simulations  and  live  players.  Point-to-point  calls  were  used 
throughout  the  duration  of  the  test  to  talk  with  individual  sites.  Problems  would  require 
that  all  participants  join  a  conference  call.  The  time  zone  differences,  along  with  the  long 
duration  of  test  events,  often  made  it  difficult  to  assemble  all  of  the  participants  during  the 
day. 

3.1.1.2  Dedicated  Voice  System 

The  ETE  and  EW  test  utilized  a  portion  of  the  available  T-l  bandwidth  to  establish 
dedicated  secure  communications  between  all  the  sites.  This  was  implemented  using  quad 
analog  voice  processor  modules  in  our  integrated  digital  network  exchange  (IDNX) 
communications  resource  managers  and  RAD  voice  signal  converters  (VSC)  to  pass  voice 
on  the  dedicated  JADS  WAN.  Each  link  had  two  voice  channels  connecting  each  site  to 
the  others.  This  reduced  the  bandwidth  available  for  data  by  128  kilobits.  The 
conferencing  capability  built  into  die  telephones  was  used  to  connect  all  sites  in  one 
conference  call.  The  “JADS  Special  Report  on  Network  and  Engineering,”  August  1999, 
has  a  more  detailed  description  of  the  IDNX  and  VSC  hardware. 

The  dedicated  voice  systems  provided  a  hot-line  capability  between  the  sites.  The 
originator  of  a  call  only  needed  to  lift  the  handset,  or  push  a  single  button  and  he  would  be 
connected  to  the  desired  site  -  usually  to  the  JADS  representative  at  the  site. 

The  second  conference  call  method  used  the  conference  feature  on  multi-line  phones 
connected  via  the  dedicated  T-l  circuits  connecting  the  facilities.  This  solution  had  the 
advantage  of  being  secure  since  all  traffic  was  sent  through  the  cryptographic  equipment 
and  used  the  JADS  WAN  links.  The  disadvantage  was  that  it  was  more  prone  to  quality 
and  procedural  problems.  This  method  was  very  sensitive  to  who  initiated  the  call,  and  it 
required  that  everyone  be  diligent  about  hanging  up  after  a  call.  The  EW  test  used  this 
method  as  the  primary  voice  system  for  test  control.  The  EW  test  used  this  method 
between  the  TCAC,  AFEWES  and  ACETEF.  At  each  of  these  facilities  this  phone 
connection  was  used  by  a  JADS  representative  to  talk  to  other  sites.  In  addition,  other 
operators  at  each  site  were  provided  with  the  ability  to  listen  in  for  commands  to  start  and 
stop  equipment  under  their  control. 

As  with  the  SIT  LSP  and  LFP,  the  EW  test  maintained  the  conference  call  throughout  the 
duration  of  test  events.  Procedures  had  to  be  developed  for  bringing  the  conference  call 
up.  The  test  director  initiated  all  calls.  In  the  case  of  problems,  all  participants  were 
required  to  hang  up  and  wait  for  the  test  director  to  initiate  a  new  call  add  sites  to  the 
conference. 


21 


In  addition  to  the  conferencing  and  dedicated  voice  requirements,  back-up  point-to-point 
communications  were  required  on  all  three  test  programs.  A  means  of  contacting  specific 
individuals  needs  to  be  defined  ahead  of  time  and  the  information  well  disseminated. 
JADS  would  prepare  telephone  rosters  before  each  event  and  give  them  to  all  personnel 
before  they  departed  for  the  field. 


3.1.1.3  Other  Considerations 

More  robust  systems  can  be  designed  but  at  significant  additional  cost.  JADS  early  design 
efforts  focused  on  expensive  systems.  Since  no  one  had  experience  in  distributed  test  and 
evaluation  (T&E),  the  requirements  were  unknown  during  the  early  test  design  phases. 
Therefore,  it  was  determined  that  it  would  be  unwise  to  design  a  costly  system  that  may 
not  satisfy  the  requirements.  The  approaches  finally  used  during  the  JADS  tests  were 
adequate  and  inexpensive  relying  on  existing  capabilities  whenever  feasible. 

3.1.2  Audio  Speakers 

To  assist  the  information  flow  within  the  TCAC,  several  test  teams  used  an  audio  speaker 
connected  to  the  main  test  control  network.  This  circuit  was  used  by  data  loggers,  analysis  and 
the  test  controller  to  track  the  approach  of  the  next  run  or  the  end  of  the  current  run.  The  test 
controller  had  the  capability  to  turn  this  speaker  off  if  needed. 


3.1.3  Headsets 

Special  headsets  were  not  required  for  the  TCAC  and  remote  data  logger  operators.  Standard 
commercial  headsets  that  connect  to  standard  telephones  were  used  at  all  sites. 

3.1.4  Voice  Recording  Capability 

Standard  VHS  video  tapes  were  used  to  record  voice  from  the  telephone  systems.  Standard 
telephones  were  rewired  to  provide  input  to  the  video  cassette  recorder  (VCR).  Two  VCRs 
allowed  two  phone  nets  to  be  recorded  for  the  SIT  LSP  and  LFP.  The  VCRs  were  connected  to 
the  Barco  video  switches  allowing  video  displays  to  be  recorded  in  addition  to  the  voice 
communications. 

3.1.5  Security 

As  with  data  networks,  voice  considerations  are  a  critical  issue  that  must  be  addressed  early  in  the 
design  process.  The  requirement  to  pass  classified  information  on  the  voice  networks  will  drive 
the  hardware  implementation  of  the  voice  network. 

Security  issues  were  a  major  concern  for  the  SIT  LFP.  Unclassified  phones  were  not  allowed  in 
many  of  the  areas  where  personnel  needed  to  be  located  during  the  test.  Classified  phone  systems 
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could  not  meet  the  requirements  of  the  test  team.  The  SIT  LFP  facilities  did  allow  us  to  use 
handsets  with  long  cords  to  extend  voice  capability  into  areas  where  phones  were  not  ordinarily 
placed.  Other  work  arounds  such  as  using  messengers  were  also  used. 

The  ETE  and  EW  tests  had  the  requirement  to  pass  classified  information  over  the  voice  network. 
A  solution  using  STU  His  would  not  satisfy  the  conference  call  requirement.  A  small  portion  of 
the  T-l  bandwidth  was  used  for  two  voice  channels.  Since  the  entire  T-l  circuit  was  encrypted, 
the  classification  requirement  was  satisfied. 


3.2  Display  Systems 

The  virtual  environment  is  often  different  from  the  one  seen  by  individual  nodes.  This  is  because 
of  the  translations  from  the  live,  virtual,  or  constructive  node  in  isolation  to  the  virtual 
environment  required  to  connect  them.  The  successful  use  of  distributed  testing  requires  a  means 
to  view  this  virtual  world.  JADS  has  used  several  viewers,  both  2D  and  3D. 

The  value  of  viewers  is  dependent  upon  the  task.  For  integration  testing  of  the  system,  both  2D 
and  3D  tools  are  used.  The  2D  viewers  can  show  the  relative  positions  of  the  entities  in  a  “God’s 
Eye”  view.  Problems  such  as  errors  in  coordinate  conversions  are  usually  easy  to  spot  with  2D 
viewers.  3-D  viewers  will  show  anomalies  such  as  missiles  coming  off  of  the  rails  backwards. 

A  3D  stealth  viewer  allows  the  operator  to  move  freely  within  the  virtual  environment  without 
creating  Protocol  Data  Units  (PDUs)  or  interacting  with  other  entities.  Viewers  without  the 
stealth  capabilities  rely  on  attaching  to  different  entities  to  see  different  portions  of  the 
environment. 


3.2.1  2D  &  3D  Displays 

The  JADS  JTF  used  several  different  2  dimensional  (2D)  and  3  dimensional  (3D)  tools  to  display 
information  used  for  test  control. 

3.2.1.1  PC  Planner  Real-time  (PLANRT) 

PLANRT  was  developed  by  NAWCWPNS,  at  Pt.  Mugu,  CA.  This  viewer  is  a  simple  2D 
viewer  but  was  the  most  effective  viewer  for  conducting  the  Systems  Integration  Test  LSP 
and  LFP  tests.  The  viewer  provided  a  simple  2D  “God’s  Eye”  representation  of  the 
entities.  Tracks  were  drawn  to  show  the  entities’  paths  over  time.  This  was  useful  in 
determining  if  the  entities  were  following  the  planned  scenario(s).  During  the  LFP  we 
found  having  a  common  view  of  the  virtual  world  was  extremely  beneficial.  The 
translation  from  real  hardware  to  the  virtual  world  can  introduce  errors  that  may  be  visible 
on  one  viewer  but  not  on  other  displays.  A  computer  with  PLANRT  installed  was  shipped 
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to  Eglin  AFB,  FL  for  use  in  integration  and  development  as  well  as  for  test  control 
purposes  during  the  LFP  tests.  This  allowed  all  participants  to  see  the  same  picture  of  the 
scenario  and  be  able  to  discuss  the  situation  within  a  common  framework. 

At  the  request  of  JADS,  NAWCWPNS  added  a  screen  to  display  target  and  shooter 
aircraft  flight  parameters  transmitted  in  data  PDUs.  Information  included  speed,  heading, 
altitude,  g-forces,  range  to  target,  and  closing  speed.  This  capability  was  used  only  during 
the  SIT  LSP  to  display  “shot  box”  parameters. 

3.2.1.2  Janus  Planview 

Janus  Planview  is  a  2D  viewer  developed  by  TRAC  WSMR.  The  software  was  designed 
for  1200  entities  and  was  not  able  to  adequately  support  10,000  entities.  The 
implementation  resulted  in  poor  refresh  rates  for  the  large  number  of  entities. 
Heterogeneous  aggregation  or  a  differing  means  of  screen  updates  are  possible  solutions. 

In  the  ETE  test,  the  heartbeat  was  turned  off  in  Janus  after  the  first  15  minutes.  This 
resulted  in  large  intervals  between  updates  for  moving  entities.  By  the  time  an  update  was 
received,  an  entity  might  be  several  kilometers  away  from  where  it  was  located  on  the 
previous  update.  Since  Planview  does  not  implement  dead  reckoning,  the  entity  locations 
of  moving  entities  were  often  incorrectly  displayed.  This  made  it  very  hard  to  follow 
moving  entities  but  locating  stationary  entities  was  very  easy. 

Planview  was  very  effective  in  providing  a  big  picture  view  of  the  battlefield.  During 
testing,  it  was  used  to  verify  that  PDUs  were  being  sent  and  that  they  were  in 
approximately  the  right  place. .  The  combination  of  a  theater  level  battlefield,  ground 
vehicles,  and  relatively  long  heartbeat  rates  did  not  provide  a  lot  of  movement  for 
observers.  During  testing,  since  the  HP735  was  utilizing  all  it’s  processing  power  to 
update  positions  it  was  virtually  impossible  to  interact  with.  A  simple  movement  of  the 
mouse  could  take  literally  minutes  to  complete.  Therefore,  the  ability  to  zoom  in  on 
specific  areas  of  the  battlefield  or  on  specific  vehicles  was  almost  impossible. 

Since  the  only  platform  it  was  tested  with  was  the  HP  735,  there  is  no  evidence  of  how  it 
would  perform  on  a  faster  platform.  The  new  series  of  HP  machines  might  be  able  to 
provide  all  the  needed  processing  power  to  keep  up  with  the  frequent  display  updates. 

3.2.1.3  Live  Entity  Viewer 

The  Live  Entity  Viewer  (LEVR),  from  TASC  is  a  3D  stealth  viewer  with  high  quality 
graphic  object  models  and  terrain  representation.  It  is  intended  for  aircraft  and  missile 
distributed  simulations.  The  version  used  by  JADS  required  a  SGI  Reality  Engine  system 
and  a  license  for  the  Paradigm  Vega  software.  TASC’s  Live  Entity  Broker  (LEB)  was 
used  to  convert  five  and  recorded  Range  Applications  Joint  Program  Office  (RAJPO)  pod 
data  to  DIS  PDUs  which  were  received  in  the  TCAC  and  displayed  using  LEVR. 
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LEB/LEVR  was  used  extensively  in  our  first  connectivity  experiment  where  JADS 
connected  to  Eglin  AFB  prior  to  the  SIT  LSP. 

3.2.1. 4  Mak  Technologies  Stealth  Viewer 

Mak  Technologies  Stealth  Viewer  is  a  3D  viewer  that  was  used  by  JADS  for  the  SIT  LSP 
and  LFP.  The  viewer  is  based  upon  Mak  Technologies  VR-LINK  product.  This  3D 
viewer  allows  people  to  see  objects  in  the  virtual  environment  in  an  easy  to  understand 
manner.  Using  this  tool,  it  is  easy  to  determine  if  the  behavior  of  entities  is  adequate.  For 
example,  this  tool  allowed  interface  developers  to  determine  that  a  simulator  to  DIS 
conversion  caused  the  target  aircraft  to  fly  upside  down.  It  is  easier  to  operate  than 
LEVR  but  lacked  the  high  resolution  terrain. 

The  usefulness  of  3D  viewer  was  limited  to  early  integration  and  development  tasks  for 
reasons  already  stated.  Therefore,  this  viewer  was  not  used  during  test  events. 

3.2.1.5  Automated  Data  Reduction  System  (ADRS) 

ADRS  was  developed  by  the  Georgia  Technical  Research  Institute  for  displaying  and 
reducing  EW  test  data.  The  visual  capabilities  include:  a  2D  display  showing  aircraft  and 
threat  missile  position  information,  a  heads-up  display  for  the  aircraft  (speed,  altitude,  and 
attitude),  a  radar  warning  receiver  display,  and  a  real-time  display  of  threat  emitter  and 
jammer  status.  The  threat  emitter  and  jammer  status  display  was  the  most  beneficial  since 
it  maHpi  it  very  clear  as  to  the  interactions  and  timing  between  the  threats  and  the  jammer. 
For  JADS,  ADRS  was  run  on  a  450  Mhz  PC.  For  performance  reasons,  two  computers 
were  required  for  test  control.  One  was  used  to  start  and  stop  the  test  and  for  the  2D 
display.  The  second  computer  displayed  the  emitter  and  jammer  status.  For  redundancy, 
a  third  computer  was  used  during  phase  2  testing  because  crashes  were  frequent. 


3.2.2  Large  Screen  Displays 

JADS  decided  that  having  information  readily  available  to  all  personnel  in  the  control  center  could 
be  beneficial.  This  would  also  allow  visitors  to  see  what  was  going  on  without  disrupting  test 
activities.  Barco  RetroGraphics  displays  were  chosen  because  they  could  operate  in  ambient 
lighting.  They  are  rear  screen  projection  devices.  Three  Barco’s  were  purchased  and  connected 
to  red,  green,  and  blue  (RGB)  video  switches.  Computers  within  the  TCAC  were  then  connected 
to  the  video  switches.  Any  connected  computer  could  be  displayed  on  any  of  the  three  Barcos. 
The  displays  could  easily  be  switched  between  computers  with  remote  controls. 

These  displays  were  most  useful  during  the  SIT  phases.  The  flight  path  information  was  useful 
and  allowed  all  personnel  to  have  a  feel  for  the  ongoing  test.  Since  the  SIT  LSP  runs  were  of 
short  duration  (approximately  2  minutes),  fast  turn  arounds  of  equipment  were  required.  All  the 
TCAC  personnel  were  able  to  track  a  run’s  progress,  giving  them  a  heads-up  about  upcoming 
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events  (e.g.,  the  end  of  the  run)  that  they  needed  to  act  upon.  The  2D  track  information,  the 
aircraft  parameters,  and  the  network  tools  were  displayed  on  the  Barcos  for  the  SIT  LSP.  The 
LFP  projected  the  2D  display,  network  tools,  and  PDU  information  from  the  STRICOM  logger 
display  on  the  barcos. 

The  display  system  was  the  least  useful  during  the  ETE  test.  The  long  duration,  slow  pace,  and 
large  number  of  entities  of  the  test  made  the  need  to  display  detailed  test  monitoring  tools 
unnecessary  and  impractical.  The  displays  were  best  used  to  give  an  easy  view  of  the  test  status 
so  large  problems  would  be  easily  noticed.  PDU  information  (total  received,  rate,  and  number  of 
entities),  Janus  PlanView,  and  Cabletron  Spectrum  (an  SMNP  tool)  were  displayed  on  the  barcos 
for  the  ETE  test. 

The  EW  test  used  the  displays  to  provide  similar  types  of  information  displayed  during  the  SIT 
phases.  A  2D  display  of  the  aircraft  and  missiles  were  projected  on  the  barcos.  Additionally, 
threat  and  jammer  states  were  displayed. 

3.3  Video  Teleconferencing 

Video  teleconferencing  (VTC)  was  originally  determined  to  be  a  JADS  requirement  based  upon 
the  work  that  was  currently  being  done  with  the  DIS  community.  The  training  community  were 
the  primary  users  of  this  technology  at  the  time  initial  requirements  were  being  formulated. 
Originally,  JADS  foresaw  benefits  from  VTC  in  the  areas  of  pre-  and  post-briefs.  As  each  test 
team  developed  detailed  requirements  it  was  determined  that  test  control  requirements  were  better 
satisfied  by  the  use  of  voice  circuits  over  the  use  of  VTC.  the  use  of  VTC  during  test  execution 
was  not  desired.. 

JADS  saw  several  problems  with  the  use  of  VTC.  The  quality  of  images  was  poor  and  most 
desktop  systems  were  designed  for  single  users.  Larger  systems  designed  for  groups  were  costly 
and  required  dedicated  circuits  and  often  special  circuits  such  as  ISDN.  ISDN  was  not  available 
to  the  JADS  facility  from  the  local  telecommunications  company.  Also,  bandwidth  consumption 
was  high  and  the  impact  on  data  transmission  appeared  to  be  significant  based  upon  theoretical 
calculations  and  lessons  learned  from  other  efforts. 

The  initial  desktop  VTC  capabilities  that  were  a  part  of  the  initial  SGI  systems  purchased  were 
deemed  insufficient  for  the  pre-brief  requirements  since  single  use  systems  cannot  be  adequately 
used  for  larger  groups.  They  had  some  utility  in  initial  network  testing  and  troubleshooting. 

Larger  VTC  systems  were  not  purchase  due  to  high  cost,  linking  requirements,  and  a  changing 
view  of  the  utility  of  such  tools.  Telephone  conference  calls  appeared  adequate  for  the  pre-brief 
and  after  action  review  requirements.  Existing  VTC  centers  at  Kirtland  AFB,  Pt.  Mugu  NAS,  and 
China  Lake  NAS  were  used  during  the  SIT  LSP  for  planning  and  reviews. 

3.4  Test  Control  and  Analysis  Center 

The  Test  Control  and  Analysis  Center  (TCAC)  was  designed  as  a  central  control  facility.  The 
TCAC  was  used  to  control  most  of  the  distributed  tests.  Analysis  of  test  data  was  the  second 
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major  function  of  the  TCAC.  Scenario  and  terrain  development  tasks  for  the  ETE  test  were  also 
accomplished  within  the  TCAC.  The  TCAC  was  designed  for  functionality  for  the  test  director 
and  computer  operators  rather  than  for  visitors  to  view  the  ongoing  tests.  Although  each  test 
established  a  different  configuration, ,  the  basic  design  was  a  U-shape  oriented  towards  the  large 
screen  displays  with  the  test  director  in  the  center.  Figure  6  shows  the  TCAC  layout  for  the  ETE 
phase  4  and  the  EW  phase  2  and  3  tests.  The  layouts  for  earlier  tests  were  similar. 

The  computer  equipment  was  positioned  to  facilitate  communications  between  individual 
operators,  the  test  director,  and  each  other.  The  primary  goal  was  to  ensure  that  the  test  director 
could  easily  talk  to  anyone  in  the  room. 

The  analysis  areas  were  separated  physically  from  the  test  control  functions.  This  allowed 
analysts  from  one  test  team  to  work  while  other  tests  were  being  conducted.  The  network  was 
also  segmented  to  prevent  traffic  from  analysis  or  other  activities  to  impact  an  ongoing  tests.  The 
ETE  test,  EW  test,  analysts,  and  networking  functions  were  all  located  on  separate  Ethernet 
segments.  A  Cisco  7000  router  was  used  to  bridge  the  four  different  Ethernet  segments.  The 
Cisco  7000  router  was  chosen  for  use  as  a  bridge  because  it  was  surplus  equipment.  Ordinarily, 
less  expensive  equipment  would  be  used  for  this  task.  This  configuration  is  shown  in  Figure  7. 
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Figure  7  TCAC  Network  Segmentation 


4.0  Test  Control  Tools 

The  ability  to  diagnose  problems  during  a  distributed  test  is  critical.  A  test  controller  needs  the 
capability  to  quickly  assess  failures  to  avoid  wasting  critical  test  resources.  JADS  used  some 
specialized  tools  to  help  monitor  different  aspects  of  the  test  such  as  the  PDU  flow  and  network 
performance  information.  This  information  proved  to  be  useful  in  finding  problems  with  the 
network  or  individual  systems  at  the  remote  nodes. 

4.1  SGI  NetVisualyzer 

SGI’s  NetVisualyzer  is  a  suite  of  tools  used  for  network  analysis.  The  tools  capture  and  examine 
network  traffic.  These  tools  are  used  during  network  setup  to  ensure  that  the  network  equipment 
are  configured  properly,  the  nodes  are  talking  to  each  other  as  intended,  and  that  extraneous 
network  traffic  is  identified  and  minimized  when  possible.  During  test  execution,  the  tools  were 
used  as  needed  to  monitor  and  trouble  shoot  the  network. 

NetVisualyzer  used  centralized  Display  Stations  and  remote  Data  Stations.  DIS  loggers  at  each 
remote  site  were  also  used  as  Data  Stations  to  collect  network  data.  The  data  from  each  Data 
Station  was  sent  to  a  Display  Station  located  in  the  Test  Control  and  Analysis  Center  (TCAC) 
where  the  information  can  be  processed  and/or  displayed. 

During  the  SIT  LSP,  the  traffic  utilization  on  local  area  networks  at  each  site  were  monitored. 
This  was  done  since  JADS  had  no  experience  in  the  utilization  rates  for  distributed  systems.  Also, 
high  traffic  rates  were  observed  at  the  F- 14  avionics  lab  in  Pt.  Mugu.  The  situation  was 
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monitored  and  the  rates  were  high  but  they  were  stable  and  did  not  impact  test  results.  NetGraph 
was  the  NetVisualyzer  tool  used  for  this  purpose. 

Provide  a  discussion  of  passive  or  active  monitoring  for  each  network  tools.  Good. 

4.2  Cabletron  SPECTRUM 

SPECTRUM®  is  a  network  analysis  package  developed  by  Cabletron  Systems.  It  provides  a  near 
real-time  capability  for  network  traffic  monitoring,  presenting  current  packet  rate  and  load 
information,  as  well  as  packet  error  and  discard  rate  information  for  network  equipment.  The 
package  also  provides  an  alarm  manager  with  simple  diagnostic  capability  that  is  valuable  in  the 
detection  and  troubleshooting  of  network  outages.  SPECTRUM®  utilizes  the  Simple  Network 
Management  Protocol  (SNMP)  to  periodically  query  network  devices  and  displays  requested 
information  on  screen  in  table  and  graphical  format.  The  SPECTRUM®  operator  can  tailor  the 
destination,  frequency,  and  content  of  the  queries  to  provide  the  desired  level  of  insight  into  a 
particular  network  portion  or  piece  of  equipment.  Like  NetVisualyzer™,  SPECTRUM®  queries 
for  data  to  create  network  traffic,  although  not  of  appreciable  quantity  to  be  noticed  in  relation  to 
the  test  traffic.  Typically,  a  five-second  polling  interval  was  used  to  monitor  the  network 
equipment,  a  value  chosen  so  that  short  duration  problem  events  would  most  likely  not  be  missed. 
Multiple  databases  store  SPECTRUM®  ’s  event  log  and  query  results  for  later  analysis. 


4.3  AG  Group,  Inc.,  EtherPeek™ 

EtherPeek™  is  a  network  analysis  package  developed  by  AG  Group  Inc.  This  product  is  a  suite 
of  protocol  analysis  tools  that  can  be  used  during  network  setup  or  during  testing  to  ensure  that 
network  equipment  is  configured  properly,  nodes  are  talking  to  one  another  as  intended,  and  to 
assist  the  network  manager  in  identifying,  eliminating,  or  minimizing  extraneous  network  traffic. 
It  provides  real-time  capability  for  network  traffic  monitoring,  presenting  current  packet  rate  and 
load  information,  as  well  as  packet  error  information.  Local  and  remote  EtherPeek™  data 
stations  passively  collect  all  LAN  traffic  at  each  site  for  real-time  network  analysis.  Unlike 
NetVisualyzer™  and  SPECTRUM®,  EtherPeek™  does  not  create  any  additional  network  traffic. 
The  protocol  analysis  part  of  the  tool  set  was  extremely  valuable  to  the  EW  Test  in  analyzing 
latency  and  data  dropouts  between  nodes. 


4.4  DIS  Loggers 

STRICOM  distributed  a  logger  in  their  DIS  Test  Suite  package  of  software.  The  logger 
displayed  PDU  information  in  a  tabular  format.  Information  such  as  instantaneous  and  average 
PDU  rates  and  the  number  of  PDUs  received  by  type  were  displayed.  While  the  logger  presented 
some  problems  recording  PDU  for  analysis  the  tool  was  useful  for  the  display.  The  displayed 
information  gave  the  test  controller  another  tool  to  verify  that  the  test  was  executing  as  expected. 
The  tool  was  not  used  to  collect  data  for  analysis  since  the  time  stamps  were  not  as  accurate  as 
JADS  required  due  to  graphic  screen  updates  interfering  with  the  time  stamp  and  recording 
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processes.  Non-graphical  loggers  were  developed  by  JADS  to  record  this  data  for  latency 
analysis. 


4.5  JADS  RTI  Interface  Logger 

This  logger  used  for  the  EW  test  resided  in  the  software  interface  between  the  federate  and  the 
RTI.  It  recorded  all  function  calls  to  and  from  the  RTI  along  with  all  the  function  data 
parameters.  For  example,  when  the  federate  wanted  to  publish  data,  it  called  the  RTI 
updateAttributeValues  function.  When  the  logger  was  linked  with  the  federate,  the  federate 
called  the  logger  updateAttributeValues  function.  The  logger  stored  the  function  identification 
and  parameter  data  in  the  log  file  buffer  and  then  called  the  RTI  updateAttributeValues  function. 
When  a  log  file  buffer  became  full,  it  was  written  asynchronously  to  the  log  file  and  a  new  buffer 
was  created. 

The  logger  was  designed  to  minimize  impact  on  the  federate  with  which  it  was  linked.  To 
accomplish  this,  the  logger  design  included  the  following  features:  asynchronous  direct  I/O, 
nondegrading  process  priority,  and  binary  file  format. 

Asynchronous  I/O  was  used  so  that  the  federate  software  did  not  wait  while  the  data  were  written 
to  the  log  file.  When  a  buffer  became  full,  an  I/O  request  was  queued  to  the  operating  system  and 
control  was  immediately  returned  to  the  federate.  A  separate  process  accomplished  the  actual 
writing  of  the  data. 

Direct  I/O  allowed  the  operating  system  to  use  the  data  buffer  created  by  the  logger  software  to 
write  the  data  to  the  disk.  Normally,  the  operating  system  copied  the  data  from  the  user  buffer  to 
a  system  buffer.  However,  use  of  direct  I/O  eliminated  this  copy  operation. 

If  a  user  with  super-user  privileges  executed  the  logger  within  the  federate,  the  logger  would  take 
advantage  of  system  nondegrading  priorities  to  further  minimize  the  impact  of  the  logger  I/O  on 
the  federate.  When  asynchronous  I/O  was  initialized,  a  set  of  processes  was  created  to  perform 
the  writing  of  the  data  to  disk.  When  these  processes  were  created,  they  inherited  the  priority  of 
the  process  that  created  them.  The  logger  software  lowered  the  priority  of  the  process  before  the 
asynchronous  I/O  was  initialized.  After  the  I/O  processes  were  created,  the  logger  software  set 
the  federate  process  priority  to  a  real-time  priority.  Since  the  I/O  processes  executed  a  lower 
priority  than  the  federate  process,  the  I/O  processes  never  interfered  with  the  federate  process. 

The  log  file  created  by  this  software  was  a  binary  file.  Attribute  and  interaction  data  were 
received  by  the  logger  (or  by  the  federate)  in  a  binary  format.  In  the  interest  of  minimizing  the 
processing  time  used  by  the  logger,  the  binary  data  received  from  or  sent  to  the  RTI  were  written 
directly  to  the  log  file  without  any  conversion. 

Since  the  logger  software  writes  all  the  binary  data  sent  to  or  received  from  the  RTI  without 
attempting  to  translate  or  convert  them,  the  logger  can  be  linked  with  any  federation  without 
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modifications  to  the  logger  software.  Also,  since  the  logger  classes  were  derived  from  the  RTI 
base  classes,  very  few  lines  of  code  must  be  changed  to  incorporate  the  logger  into  an  existing 
federate.  Less  than  twenty  lines  of  code  were  modified  or  added  to  link  the  logger  with  the 
helloWorld  demo  program  provided  with  the  RTI. 

4.6  HLA  Link  Health  Check 

A  combination  of  in-house  tools  and  commercially  developed  software  products  provided  JADS 
with  a  real-time,  or  near  real-time,  limited  capability  to  assess  network  performance  and  evaluate 
the  integrity  of  data  as  they  were  being  collected  during  Phase  3  testing.  The  various  tools  were 
used  to  help  provide  a  clear  picture  of  the  network  and  to  speed  diagnostic  and  maintenance 
efforts  during  a  test.  Simple  Network  Management  Protocol  (SNMP)  tools  were  used  to  monitor 
communications  and  network  hardware.  This  allowed  JADS  personnel  to  see,  in  near  real  time, 
the  status  of  the  long-haul  links  as  well  as  the  routers  connecting  remote  sites.  The  SNMP  tools 
also  allowed  JADS  to  monitor  and  record  the  bandwidth  used  on  the  T-l  links. 

Each  federate  sent  link  health  check  updates  and  displayed  the  information  received  from  other 
federates.  This  was  used  during  testing  not  only  to  determine  the  health  of  the  each  federate  but 
as  another  view  of  the  overall  health  of  the  network.  In  addition,  a  simple  UNIX  utility  used 
standard  pings  to  display  the  status  of  various  federation  computers.  This  tool  often  presented  the 
test  team  and  networking  personnel  with  the  first  indication  that  a  problem  existed  with  the 
network  and/or  the  computers  at  each  site. 


The  JADS  EW  test  required  all  federates  to  publish  a  link  health  check  message  once  per  second. 
The  federates  developed  by  Georgia  Tech  Research  Institute  included  displays  of  information 
from  the  link  health  check  messages  received  from  each  of  the  federations.  This  information 
displayed  in  the  TCAC,  was  used  by  the  test  controller  to  help  control  the  sequence  and  timing  of 
the  federation  startup  for  each  run.  As  each  run  executed,  the  displays  also  alerted  the  test 
controller  and  federate  operators  to  problems  with  specific  federates.  If  messages  from  a 
federate  were  no  longer  being  received  this  was  displayed.  It  could  then  be  determined,  usually 
by  voice,  whether  the  federate  had  crashed  or  another  problem  (such  as  with  the  network)  had  to 
be  investigated. 

4.7  Time  Synchronization 


JADS  used  both  software  and  hardware  methods  to  synchronize  the  computers  across  the 
distributed  test  network.  The  software  solution  was  required  because  the  SGI  workstations  had 
proprietary  bus  architectures  and  time  cards  were  not  available  for  that  bus  early  in  the  program. 
A  Global  Positioning  System  (GPS)  receiver  was  used  to  provide  Inter-range  Instrumentation 
Group  (IRIG-B)  time  signals  to  a  single  SGI  via  serial  cable.  The  SGI  became  a  stratum  one  time 
server  and  a  network  time  protocol  application,  XNTP,  from  the  University  of  Delaware  was  used 
to  distribute  time  to  all  of  the  SGI  and  Sun  workstations  throughout  the  network.  One  step  in  the 
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network  startup  procedures  before  tests  required  that  the  time  synchronization  be  tested.  The 
XNTP  software  has  built  in  tools  that  allow  comparisons  between  the  time  on  two  machines. 

This  allowed  the  time  on  each  remote  workstation  to  be  compared  with  the  stratum  one  server  to 
determine  accuracy. 


The  hardware  solution  was  used  for  the  EW  test.  By  the  time  we  began  to  select  and  integrate 
hardware  for  the  EW  test,  time  cards  for  the  SGI  and  Windows  NT  computers  were  available. 
Each  computer  had  a  Bancomm  time  card  from  Datum.  Each  of  the  three  sites  were  equipped 
with  GPS  receivers  which  became  IRIG-B  time  sources  for  the  cards. 

The  ability  to  compare  and  synchomize  time  between  the  facilities  was  not  available  with  the 
hardware.  Time  synchronization  was  performed  by  initiating  a  normal  run  and  after  a  jamming 
response  was  observed  in  the  TCAC  the  execution  was  stopped.  The  recorded  times  for  all 
messages  in  federate  log  files  were  then  compared.  If  they  were  “reasonable”  (within  +/-  20 
milliseconds  of  each  other),  the  facilities  were  considered  to  be  in  synchronization.  The  inability 
to  confirm  synchronization  and  monitor  it  throughout  the  test  might  be  considered  a  limitation  of 
the  hardware  solution.  Regardless  of  the  trust  we  place  in  GPS  as  a  time  source,  the  JADS  test 
data  did  identify  at  least  one  instance  where  a  facility’s  timing  drifted  out  of  synchronization. 


4.8  JADS  Toolbox 

JADS  discovered  early  on  that  the  test  controller  and  analysts  needed  displays  for 
troubleshooting,  analysis,  and  visualization.  To  meet  these  needs,  JADS  developed  a  software 
product  for  JADS  called  the  JADS  Analysis  Toolbox.  It  comprised  a  set  of  analysis  tools 
integrated  into  a  single  user  interface.  It  allowed  users  to  view  protocol  data  unit  (PDU)  data  in 
near  real  time,  replay  PDUs  from  a  log  file,  post-test,  and  obtain  various  PDU  data/statistics, 
post-test.  The  toolbox  was  used  for  quick-look  analysis  and  monitoring  of  test  data  throughout 
the  tests. 

The  JADS  Analysis  Toolbox,  developed  for  the  SIT,  worked  very  well  for  the  SIT.  However, 
without  modifications,  it  was  inadequate  for  the  ETE  test.  Long  delays  were  introduced  because 
of  the  large  size  of  the  ETE  test  log  files.  In  addition,  the  number  and  method  of  creating  entities 
for  the  ETE  test  were  significantly  different  from  the  method  used  for  the  SIT.  This  required 
changes  in  the  way  analysis  routines  handled  data  from  the  ETE  test  files. 

5.0  Test  Control  Lessons  Learned 

These  lessons  learned  incorporate  findings  from  interim  and  final  reports  for  SIT,  ETE,  and  EW 
test  efforts. 


5.1  Planning- 
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The  test  control  requirements  for  an  ADS  test  must  be  clearly  defined  early  in  the  test 
planning  phase. 

—  Detailed  planning  and  coordination  are  required  to  ensure  a  common  understanding  of 
all  requirements,  procedures,  and  test  objectives.  Individual  facilities  are  not  generally 
familiar  with  conducting  coordinated,  distributed  T&E  tests  and  this  common 
understanding  is  critical.  Test  control  must  be  an  early  part  of  the  planning  process  to 
avoid  last  minute  control  issues  between  different  nodes. 

Piggybacking  off  of  other  activities  is  not  always  practical.  Some  attempts  were  made  to 
accomplish  SIT  LFP  objectives  by  leveraging  off  of  other  activities  at  Eglin,  but  these 
piggyback  attempts  resulted  in  significant  delays  and  problems  satisfying  SIT  LFP 
requirements.  These  attempts  were  unsuccessful  because  of  conflicts  in  test  control  of  the 
piggyback  flight  assets  and  support  equipment. 


Configuration  control  is  essential.  Configuration  control  of  the  network  and  the  ADS/DIS 
system,  including  its  hardware,  software,  and  its  simulator  interfaces,  is  necessary  starting 
at  the  beginning  of  the  program..  It  is  particularly  important  during  test  events  that  the  test 
controller  is  aware  of  configuration  changes  and  that  they  are  approved  prior  to  changes. 


Pilots  should  be  involved  in  scenario  setup.  It  was  very  beneficial  to  contact  the  test 
squadron  pilots  in  order  to  achieve  the  correct  flight  profiles  and  launch  parameters.  The 
LFP  missions  were  based  on  previously  flown  AMRAAM  profiles,  and  the  goal  was  to 
replicate  those  profiles  (within  LFP  limitations).  It  was  difficult  to  match  the  scenario 
launch  conditions  during  high-g  maneuvers.  The  best  results  came  when  the  project 
personnel  met  with  the  pilots  to  review  possible  setup  options,  then  reviewed  the  flight 
plan  during  the  pilots’  pre-mission  briefing. 

Live  missions  require  more  test  control  contingency  planning.  Live  aircraft  operations 
required  more  contingency  planning  to  quickly  decide  on  alternatives.  A  great  benefit  for 
using  an  ADS-linked  network  was  having  several  “analysts-in-the-loop”  during  the  real¬ 
time  mission.  The  JADS  test  director  had  to  make  timely  decisions  between  the  aircraft 
passes  in  order  to  affect  the  outcome  or  productivity  of  the  mission.  Initially,  a  small  list 
of  go/no  go  criteria  was  made  before  the  first  live  flight.  This  list  included  criteria  on  key 
aircraft  components,  and  on  aircraft  pods  or  ground  systems.  As  the  experience  level 
increased  after  risk  reduction  missions,  this  list  was  expanded  to  include  alternatives  in 
case  of  failures  or  degraded  assets.  Having  this  contingency  plan  spelled  out  in  advance 
truly  helped  the  test  director  make  rapid,  well-informed  decisions  to  get  the  most 
productive  use  out  of  the  remaining  mission  time. 

Use  risk  reduction  tests  for  integration  and  test  control  procedures  The  series  of  risk 
reduction  tests  was  extremely  useful  to  progress  through  many  levels  of  integration  work. 
The  Eglin  members  designed  a  building  block  approach  to  check  out  interfaces  at  the 
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lowest  level,  as  well  as  developing  test  control  procedures.  The  risk  reduction  efforts 
allowed  test  procedures  to  be  refined  prior  to  actual  test  events. 

High  personnel  turnover  hurts  test  control  development.  Not  surprisingly,  high  turnover 
made  the  development  of  procedures  and  training  of  test  participants  difficult.  Some  of  the 
JADS  test  efforts  were  conducted  over  lengthy  time  spans.  From  one  test  phase  to  the  next 
changes  in  operators  made  the  use  of  rehearsals  even  more  critical  to  successful  testing.  All 
three  of  the  JADS  test  experienced  the  loss  of  key  personnel  over  the  test  life  spans.  The 
distributed  nature  of  the  effort  did  not  allow  for  tight  personnel  controls  possible  at  a  single 
site. 


5.2  Security 

—  Network  security  is  an  essential  part  of  operating  an  ADS  network.  The  network  accreditation 
process  needs  to  be  addressed  and  started  in  the  planning  phase.  Security  and  accreditation 
procedures  are  different  for  every  site  and  branch  of  service.  Hence,  it  becomes  inherently 
difficult  to  execute  inter-agency  memorandums  of  agreement  (MO A).  Security  focal  points 
and  designated  approval  authorities  need  to  be  identified  early  in  the  planning  process  and  close 
coordination  with  these  individuals  is  essential  to  executing  network  installation  and  meeting 
test  schedules.  It  is  wise  to  factor  approximately  three  months  into  the  implementation 
schedule  to  execute  the  required  security  MO  As. 


—  OPSEC  requirements  for  a  distributed  weapon  system  test  must  be  determined  and 
coordinated  early  in  the  program,  especially  with  the  various  organizations  and  their  different 
procedures.  Issues  to  be  addressed  include:  the  need  for  an  OPSEC  Plan;  if  an  OPSEC  Plan 
is  required,  an  agreement  either  to  use  an  already  approved  OPSEC  Plan  or  to  draft  a  new 
OPSEC  Plan;  the  consistency  of  OPSEC  requirements  among  the  various  organizations  and 
programs;  OPSEC  requirements  in  test  control/conduct,  including  the  use  of  “For  Official  Use 
Only”  test  cards  and  step  calls  and/or  the  use  of  secure  communications. 


5.3  Network  Instrumentation  Tools 


-  Time  sources  must  be  synchronized.  IRIG/GPS  time  must  be  synchronized  off  of  the 
same  time  source  and  then  must  be  validated  at  each  test  site  prior  to  project  operations  to 
ensure  accurate,  synchronized  time  is  precisely  recorded  at  each  test  site.  A  test  controller 
needs  this  common  time  reference  to  control  and  trouble  shoot  problems  which  occur 
during  test  events.  The  level  of  accuracy  will  vary  depending  on  the  test  but  easily  could 
vary  from  one  millisecond  to  several  seconds. 
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-  Special  test  equipment  is  needed  for  checkout  and  verification.  Special  test  equipment 
(e.g.,  SNAP  loggers,  DIS  data  loggers,  etc.)  and  other  networking  tools  (e.g.,  Stealth 
Observer)  should  be  part  of  each  simulation  node’s  configuration  during  the  development, 
test,  checkout,  and  verification/validation  phases  in  all  subsequent  ADS  testing.  Special 
test  equipment  and  networking  tools  are  required  to  more  rapidly  isolate  and  determine 
the  specific  cause  of  network,  ADS/DIS,  etc.  problems.  This  capability  is  a  key  part  of 
effective  test  control.  If  problems  can  not  be  diagnosed  quickly  a  test  controller  may 
waste  the  availability  of  critical  test  assets. 


5.4  Command  and  Control 

—  Have  a  centralized  test  control  center.  Test  controllers  should  be  extremely  familiar  with  the 
test  and  network  configuration.  The  JADS  TCAC  was  configured  to  allow  for  convenient, 
instant  communications  with  all  the  nodes.  It  acted  as  the  central  point  of  contact  between  the 
nodes  and  for  all  problems.  The  test  controller  kept  track  of  test  progress,  documented  any 
problems  that  occurred  and  kept  the  test  director  informed. 

-  Project  and  scenario  control  were  best  performed  at  the  test  range.  Another  issue  was  where 
to  locate  project  decision  makers  to  control  the  scenarios  and  test  events.  Dining  early  risk 
reduction  tests,  it  became  apparent  that  when  handling  developmental  problems,  there  was 
significant  discussion  that  was  never  transmitted  via  radio  links.  Much  troubleshooting  was 
still  communicated  face-to-face,  over  local  intercom,  or  via  direct  phone  line.  Even  with  some 
display  monitors,  personnel  at  remote  sites  would  often  hear  about  problems  late,  often  with 
only  partial  explanations.  Therefore,  in  order  to  be  part  of  the  decision  making  process,  the 
JADS  test  director  had  to  be  collocated  with  the  CCF  coordinator  and  the  range  aircraft 
controller.  The  CCF  was  the  necessary  choice,  since  it  was  centrally  located  with  the  critical 
personnel,  the  majority  of  system  displays,  all  communication  networks  including  a  direct  link 
to  the  test  controller  in  the  TCAC. 


-  Distributed  tests  require  personnel  distribution.  When  many  distributed  nodes  are  required  for 
the  successful  completion  of  a  test,  personnel  will  need  to  be  located  at  these  nodes.  The 
complexity  and  input  an  individual  node  contributes  should  guide  the  assignment  of  personnel. 

A  T-24  hours  pre-mission  briefing  is  needed  before  each  mission.  A  pre-mission  briefing  was 
held  24  hours  before  each  mission  and  was  critical  for  coordinating  the  many  network  and 
flight  test  issues.  The  briefing  agenda  by  the  test  director  should  include  the  test  objectives, 
planned  run  matrix,  personnel  involved,  communication  plan  with  back-up  phone  numbers, 
show  time  for  personnel,  go/no  go  criteria,  contingency  plans  in  case  of  failures, 
instrumentation  and  data  collection  requirements,  and  details  on  facility  configuration. 

Day  of  test  briefings  are  needed  before  and  after  each  mission.  Briefs  and  debriefs  should  be 
conducted  before  and  after  each  mission.  The  briefs  should  cover  such  items  as  the  test 
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objectives,  telephone  numbers/frequencies  to  use  for  test  control,  test  configuration  of  each 
facility,  instrumentation  and  data  collection  requirements,  go/no  go  criteria,  contingency/back¬ 
up  plans,  test  conduct  including  a  detailed  review  of  test  cards,  communications  procedures, 
OPSEC,  and  the  time  and  place  of  the  debrief.  A  briefing  checklist  should  be  developed  and 
used  by  the  test  director. 

Adequate  time  must  be  allotted  for  data  analysis  between  test  events.  There  was  a  tendency 
to  underestimate  the  time  required  to  adequately  analyze  the  large  volume  of  data  collected  in 
the  test  events.  As  a  result,  some  problems  from  one  mission  were  not  fully  diagnosed  and 
fixed  before  the  next  mission.  In  fact,  some  problems  (e.g.,  target  initialization  errors)  were 
not  even  recognized  until  all  test  was  over.  Rehearsal  of  the  analysis  procedures  should  be 
used  to  better  estimate  the  time  required  for  adequate  analysis  between  test  events. 


Tight  control  of  the  aircrew  is  not  desirable.  The  aircrew  should  be  allowed  to  perform  as 
aircrew  during  man-in-the-loop  testing.  Too  much  test  control  (e.g.,  “fire”  instead  of  “cleared 
to  fire”)  is  not  desirable  with  the  man-in-the-loop,  particularly  if  it  is  OT&E  or  combined 
DT&E/OT&E.  Testing  is  more  valuable  and  there  are  more  “lessons  learned”  from  a  test 
where  the  aircrew  are  given  the  critical  parameters  and  switchology  to  meet  the  test 
objective(s)  and  are  still  allowed  to  make  tactical  decisions,  fly  the  “aircraft,”  operate  the 
weapon  system,  etc.  A  tightly  controlled  test  is  appropriate  for  certain  testing  such  as 
computer  simulations  and  stand-alone  laboratory  tests. 

Additional  time  is  needed  before  the  beginning  and  after  the  end  of  each  testing  period. 
Allocate  a  minimum  of  an  additional  two  hours  of  time  at  the  end  of  each  test  period  for  data 
logging,  data  archiving,  data  transfer,  and  laboratory  reclassification.  Allocate  a  minimum  of 
an  additional  one  hour  of  set-up  time  prior  to  each  test  period.  The  pre-  and  post-test 
requirements  should  be  included  in  the  number  of  hours  needed  for  each  test  period  and 
incorporated  into  the  planned  costs.  This  additional  time  must  be  coordinated  between  all 
nodes  by  the  test  controller. 


Test  control  procedures  should  be  well  rehearsed.  Communication  and  coordination  among 
ADS  test  team  members  are  vital  for  the  success  of  an  ADS  test  especially  when  changes  or 
additions  are  made  to  the  test  environment.  When  many  people  are  communicating  on  one 
phone  line,  a  response  order  should  be  established  and  strictly  followed  to  save  valuable  test 
time. 

Two-dimensional  displays  were  needed  at  each  node.  Using  a  graphical  2-D  display  greatly 
improved  the  situational  awareness  of  participants  at  each  testing  site.  The  2-D  display 
converted  the  ES  PDUs  into  symbols  overlaid  on  a  local  area  map.  As  a  result,  the  team 
members  knew  where  and  in  what  direction  the  aircraft  were  flying  and  if  the  missile 
simulation  was  active. 
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Existing  range  procedures  had  to  be  modified  for  ADS  The  existing  Eglin  test  procedures 
were  only  written  for  individual  facilities,  so  a  new  combined  checklist  was  created  for  ADS 
applications.  The  AFDTC  Program  Manager  developed  a  new  checklist  which  interleaved 
some  actions  from  each  facility  in  order  to  get  proper  connectivity  between  the  CCF, 
MISILAB,  and  PRIMES.  This  checklist  covered  activities  from  24  hours  prior,  4  hours  prior,' 
1  hour  prior,  and,  of  course,  during  the  mission  time. 

Lab  replays  served  as  an  excellent  method  of  rehearsal.  By  replaying  a  set  of  prerecorded 
data,  all  the  Eglin  team  members  could  participate  in  a  laboratory  replay  session.  This  was  an 
excellent  method  to  rehearse  test  procedures,  work  out  technical  and  procedural  issues,  and 
troubleshoot  problems  in  a  low-cost,  low-stress  lab  session. 


5.5  Communications 

'  Igst  Control  communications  requirements  must  be  addressed  early  in  the  test  planninp 

This  is  necessary  to  ensure  effective  communications  during  the  test.  Remote  test 
control  for  SIT  test  used  two  non-secure  telephone  conference  bridges  (i.e.,  two 
communications  nets)  which  were  acceptable.  However,  the  audio  level  between  connections 
varied,  making  “loud  and  clear”  communications  among  all  the  sites  difficult  at  times.  ETE 
and  EW  testers  also  used  dedicated  voice  circuits  on  segmented  portions  of  the  network  T-l 

lmes.  If  T-l  lines  are  to  be  used  for  test  control  then  planning  should  include  the  120  days 
lead  time  needed  to  install  them. 

A  standard,  linked  test  should  have  multiple  (more  than  two)  communications  nets  (e.g., 
control,  analyst,  network,  and  internal  laboratory)  with  easy,  selectable  access  to  all  the  nets 
from  multiple  locations  within  the  laboratory.  A  minimum  of  one  secure  telephone  at  each  site 
is  also  required.  More  complex,  linked  tests  may  require  additional  non-secure  and/or  secure 
communications  nets.  Speakers  with  plug-in  push-to-talk  handsets  and/or  headsets  with  push- 
to-talk  switches  are  required  at  strategic  locations  throughout  the  laboratory  to  allow 
individuals  the  necessary  mobility  and  communications  flexibility.  Speakers  are  also  necessary 
when  VIP  visitors  or  other  personnel  monitor  the  test  at  the  laboratory. 

-  The  capability  for  secure  video  teleconferences  (VTCs)  among  multiple  (more  than  two)  sites 
is  helpful  for  pre  and  post  test  mission  briefs.  Additionally,  secure  telephone  conference 
bridges  are  also  required  when  a  test  must  be  conducted  using  secure  communications  (e.g., 

a  special  project  test),  or  when  a  secure  VTC  is  not  available  and  secure  briefs/debriefs  are 
necessary. 

~  Thoroughly  test  the  voice  communication  systems  followed  by  a  rehearsal  with  all  sites  and  the 
actual  players  before  any  test  event.  It  is  critical  that  the  personnel  involved  in  a  test 
participate  in  any  rehearsals  so  that  they  fully  understand  the  processes  and  procedures, 
including  contmgency  plans.  Backup  means  of  communication  are  critical.  Point-to-point 
telephones  and  beepers  were  the  primary  backup  methods  used  by  JADS.  At  times,  even 
faxes,  computer  talk  sessions,  and  e-mail  were  used. 
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—  Security  issues  can  affect  how  voice  communications  are  implemented  in  individual  facilities. 
Facility  security  policies  often  determine  where  phones  may  or  may  not  be  placed.  The 
restrictions  may  be  based  on  keeping  phones  away  from  electronic  emanations  from  classified 
systems  or  so  that  classified  conversations  may  not  be  inadvertently  passed  across  an 
unclassified  phone  system.  Each  facility  and  each  test  program  has  to  be  handled  differently; 
the  important  thing  to  remember  is  to  determine  the  restrictions  early  enough  for  work¬ 
arounds  to  be  determined  and  implemented. 
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AFATDS 

AFB 
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ATACMS 
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DIS 

DMAP 

DMSO 

DoD 

DSI 

DSM 

DSU 

DT&E 

ECM 

EPF 

ES 

ESPDU 

ETE 

EW 


analysis  and  control  element 

Air  Combat  Environment  Test  and  Evaluation  Facility,  Patuxent  River, 
Maryland;  Navy  facility 

Advanced  Distributed  Electronic  Warfare  System;  Army  sponsored 
advanced  distributed  simulation 
air  data  terminal 

Advanced  Field  Artillery  Tactical  Data  System 
Air  Force  Base 

Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas;  Air 
Force  managed  with  Lockheed  Martin  Corporation 
air  intercept  missile 

a  mature  self-protection  jammer  system;  an  electronic  countermeasures 
system  with  reprogrammable  processor  developed  by  Georgia  Tech 
Research  Institute 

advanced  medium  range  air-to-air  missile 

application  program  interface 

Advanced  Radar  Imaging  Emulation  System 

All  Source  Analysis  System 

Army  Tactical  Missile  System 

Aviation  Test  Bed  at  Fort  Rucker,  Alabama 

Battle  Management  Interoperability  Center  at  Naval  Air  Warfare  Center, 
Point  Mugu,  California 

command,  control,  communications,  computers  and  intelligence 
command,  control,  communications,  computers,  intelligence,  surveillance 
and  reconnaissance 

Central  Control  Facility,  Eglin  Air  Force  Base,  Florida 
common  ground  station 
concept  of  operations 

Office  of  the  Secretary  of  Defense  committee  under  the  director,  Test, 

Systems  Engineering  and  Evaluation 

channel  service  unit 

distributed  interactive  simulation 

data  management  and  analysis  plan 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

Department  of  Defense 

Defense  Simulation  Network 

digital  system  model 

data  service  unit 

developmental  test  and  evaluation 
electronic  countermeasures 
engineering  protofederation 
electronic  support 
entity  state  protocol  data  unit 
JADS  End-to-End  Test 

electronic  warfare;  JADS  Electronic  Warfare  Test 


FAT 

federate  acceptance  test 

FBCB2 

Force  XXI  Battle  Command,  Brigade  and  Below 

FED 

federation 

FEDEP 

federation  development  and  execution  process 

FEDEX 

federation  executive 

FIT 

federate  integration  test 

FOM 

federation  object  model 

FOT&E 

follow-on  operational  test  and  evaluation 

FTP 

file  transfer  protocol 

FY 

fiscal  year 

GB 

gigabyte 

GDT 

ground  data  terminal 

GPS 

global  positioning  system 

HITL 

hardware-in-the-loop  (electronic  warfare  references) 

HLA 

high  level  architecture 

HW 

hardware 

HWIL 

hardware-in-the-loop  (system  integration  references) 

IADS 

Integrated  Air  Defense  System 

ICD 

interface  control  document 

ID 

infantry  division;  identification 

IDNX™ 

Integrated  Digital  Network  Exchange 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IGMP 

Internet  Group  Management  Protocol 

INS 

inertial  navigation  system 

IP 

Internet  protocol 

IPPD 

integrated  product  and  process  development 

IPT 

integrated  product  team 

IR 

infrared 

IRIX 

operation  system  for  the  Silicon  Graphics,  Inc. 

ISTF 

installed  systems  test  facility 

IRIG 

Inter-Range  Instrumentation  Group 

J/S 

jamming-to-signal  ratio 

JADS 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

Janus 

interactive,  computer-based  simulation  of  combat  operations 

JCSAR 

Joint  Combat  Search  and  Rescue 

JECSIM 

Joint  Electronic  Combat  Testing  Using  Digital  Simulations 

JETS 

JammEr  Techniques  Simulator 

JSF 

Joint  Strike  Fighter 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

JT&E 

joint  test  and  evaluation 

JTF 

joint  test  force 

JTMD 

Joint  Theater  Missile  Defense 

LAN 

local  area  network 

LFP 

Live  Fly  Phase 

LGSM 

light  ground  station  module 
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LHC 

LRC 

LSP 

M&S 

Mbps 

MCTS 

MISILAB 

MITRE 

MOE 

MOP 

MOT&E 

MSL 

NATO 

NETV  isualizer™ 

NIU 

NTP 

OAR 

OPTEMPO 

OSD 

OT 

OT&E 

OTA 

PC 

PDU 

PGM 

PIM-DM 

PMO 

P-value 

RDAPAS 

RDL 

RELDISTR 

RF 

RFENV 

RID 

RM&A 

RTI 

RTIEXEC 

SAIC 

SAR 

SATCOM 

SBA 

SCDL 

SE 

SEOT 


link  health  check 

local  runtime  infrastructure  component 

Linked  Simulators  Phase 

modeling  and  simulation 

megabits  per  second 

Mission  Crew  Training  System 

Missile  Simulation  Laboratory,  Eglin  Air  Force  Base,  Florida 

company  that  provided  engineering  services 

measure  of  effectiveness 

measure  of  performance 

multiservice  operational  test  and  evaluation 

missile 

North  Atlantic  Treaty  Organization 

software  that  displays  real-time  bandwidth  use  in  a  rolling  bar  graph  format 

for  quick  visual  reference 

network  interface  unit 

network  time  protocol 

open  air  range 

operations  tempo 

Office  of  the  Secretary  of  Defense 

operational  test 

operational  test  and  evaluation 

Office  of  Technology  Assessment 

personal  computer 

protocol  data  unit 

precision  guided  munitions 

protocol  independent  multicast-dense  mode 

program  management  office 

probable  value 

Radar  Detection  and  Performance  Analysis  System 

rear  data  link 

reliable  distribution 

radio  frequency 

radio  frequency  environment 

runtime  infrastructure  initialization  data 

reliability,  maintainability  and  availability 

runtime  infrastructure 

runtime  infrastructure  executive 

Science  Applications  International  Corporation 

synthetic  aperture  radar 

satellite  communications 

Simulation  Based  Acquisition 

surveillance  control  data  link 

synthetic  environment 

synthetic  environment  operational  test 
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SETI-VTP 

Synthetic  Environment  Tactical  Integration 

VTP 

Virtual  Torpedo  Project 

SGI 

Silicon  Graphics,  Inc. 

SIL 

system  integration  laboratory 

SIM 

simulation 

SIMLAB 

Simulation  Laboratory,  Naval  Air  Warfare  Center,  China  Lake,  California 

SINCGARS 

Single-Channel  Ground  and  Airborne  Radio  System 

SIT 

JADS  System  Integration  Test 

SMC 

source  mode  change 

SME 

subject  matter  experts 

SMS 

stores  management  system 

SOW 

statement  of  work 

SPECTRUM® 

a  network  analysis  package  developed  by  Cabletron  Systems 

SPJ 

self-protection  jammer 

SRS 

software  requirements  specification 

STEP 

simulation,  test  and  evaluation  process 

STORM 

Simulation,  Testing  and  Operations  Rehearsal  Model 

STTAR 

synthetic  test  and  training  architecture 

SUT 

system  under  test 

SW 

software 

T&E 

test  and  evaluation 

T/E 

tracking  error 

T-l 

digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1 .544  megabits 
per  second 

T-3 

298  T-l  lines  in  one;  the  aggregate  data  rate  is  44.746  megabits  per  second 

TAC 

target  analysis  cell 

TACCSF 

Theater  Air  Command  and  Control  Simulation  Facility 

TAFSM 

Tactical  Army  Fire  Support  Model 

TAMS 

Tactical  Air  Mission  Simulator 

TCAC 

Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 

TCF 

test  control  federate 

TCP 

transmission  control  protocol 

TDP 

time-space-position  information  data  optimizingprocessor 

TEMP 

test  and  evaluation  master  plan 

TGT 

target 

TMD 

Theater  Missile  Defense 

TRAC 

U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TSLA 

Threat  Simulator  Linking  Activity 

TSPI 

time-space-position  information 

TTH 

terminal  threat  hand-off  federate 

TTP 

tactics,  techniques  and  procedures 

UDP 

user  data  protocol 

UHF 

ultra  high  frequency 

UMB 

umbilical 
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UNIX™ 

registered  trademark  of  UNIX  Systems  Laboratories 

V&V 

verification  and  validation 

VPG 

virtual  proving  ground 

VSTARS 

Virtual  Surveillance  Target  Attack  Radar  System 

W&A 

verification,  validation,  and  accreditation 

WAN 

wide  area  network 

WBS 

work  breakdown  structure 

WSIC 

Weapons  System  Integration  Center,  Naval  Air  Warfare  Center,  Point 
Mugu,  California 

WSMR 

White  Sands  Missile  Range,  New  Mexico 

WSSF 

Weapon  System  Support  Facility,  China  Lake,  California 
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